Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
e17b53a
fix: adjust test case to start with
dlaehnemann Jan 26, 2026
390c361
fix: missing_run_exports linting checks either all `outputs:` or only…
dlaehnemann Jan 26, 2026
142e197
docs: adjust lint explanation to reflect outputs structure
dlaehnemann Jan 26, 2026
d1aefb7
test: remove build number in outputs, it only makes sense on the glob…
dlaehnemann Jan 27, 2026
20dff6a
test: also remove build sections under outputs for empty_build_sectio…
dlaehnemann Jan 28, 2026
ccff0ad
fix: lint for no `requirements:` on top level if `outputs:` specified
dlaehnemann Jan 29, 2026
6b1eb4b
fix: working outputs-specific solution for `should_be_noarch_generic`…
dlaehnemann Jan 29, 2026
603921d
fix: do check_deps() per output, if `outputs:` specified, and include…
dlaehnemann Jan 30, 2026
d0d0ef7
fix: should_be_noarch_generic now works with the new per-output setup
dlaehnemann Jan 30, 2026
60b51eb
fix: attempt of introducing an `extra: skip-recipes:` mechanism, but …
dlaehnemann Jan 30, 2026
910c463
format with ruff
johanneskoester Feb 4, 2026
3eeab8e
Revert "format with ruff"
johanneskoester Feb 4, 2026
498191a
Merge branch 'master' into pr/dlaehnemann/1073
johanneskoester Feb 4, 2026
b2ee36f
fmt
johanneskoester Feb 4, 2026
438f1b9
fmt
johanneskoester Feb 4, 2026
a8f9275
fix: generalize disallowed top-level sections lint for when `outputs:…
dlaehnemann Feb 5, 2026
4dbff8d
fix: add `package_location` to `check_deps()` arguments consistently
dlaehnemann Feb 5, 2026
dfa26c0
Merge branch 'fix/adjust-missing-run-exports-lint-to-new-outputs-stru…
dlaehnemann Feb 5, 2026
0bdd2a9
fix: Update anaconda-client version to 1.14.* (#1075)
fxwiegand Feb 11, 2026
e358b39
update action path
bgruening Feb 11, 2026
c77bc5e
Update release-please.yml
bgruening Feb 11, 2026
e301b5c
fix: update actions (#1076)
bgruening Feb 11, 2026
975acdb
chore(master): release 4.0.0 (#1077)
github-actions[bot] Feb 11, 2026
3d4c9e1
feat: eliminate redundant host-side solver run for Docker builds (#1081)
nh13 Mar 2, 2026
af1ac9c
feat: pre-solved environments for mulled tests (#1082)
nh13 Mar 3, 2026
2f558ee
fix: add raise_for_status to CircleCI workflow API call (#1083)
nh13 Mar 4, 2026
992d5c3
chore(master): release 4.1.0 (#1084)
github-actions[bot] Mar 4, 2026
0cbea1b
test: overhaul lint testing with one yaml per test (#1085)
dlaehnemann Mar 26, 2026
ef5af0f
Merge branch 'fix/adjust-missing-run-exports-lint-to-new-outputs-stru…
dlaehnemann Mar 30, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions .github/workflows/changevisibility.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ jobs:
run:
shell: bash -l {0}
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6

- name: Check Containers and Set Public
run: |
Expand All @@ -20,8 +20,9 @@ jobs:
QUAY_OAUTH_TOKEN: ${{ secrets.QUAY_BIOCONTAINERS_TOKEN }}

- name: Upload logs
uses: actions/upload-artifact@v4
uses: actions/upload-artifact@v6
with:
name: logs
path: biocontainers-*.txt
retention-days: 7

5 changes: 3 additions & 2 deletions .github/workflows/release-please.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@ on:
push:
branches:
- master
workflow_dispatch:

name: release-please

Expand All @@ -12,7 +13,7 @@ jobs:
release_created: ${{ steps.release.outputs.release_created }}
tag_name: ${{ steps.release.outputs.tag_name }}
steps:
- uses: GoogleCloudPlatform/release-please-action@v4
- uses: googleapis/release-please-action@v4
id: release
with:
release-type: python
Expand All @@ -23,7 +24,7 @@ jobs:
needs: release_please
if: needs.release_please.outputs.release_created
steps:
- uses: actions/checkout@v4
- uses: actions/checkout@v6
with:
fetch-depth: 0

Expand Down
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,31 @@
# Changelog

## [4.1.0](https://github.com/bioconda/bioconda-utils/compare/v4.0.0...v4.1.0) (2026-03-04)


### Features

* eliminate redundant host-side solver run for Docker builds ([#1081](https://github.com/bioconda/bioconda-utils/issues/1081)) ([3d4c9e1](https://github.com/bioconda/bioconda-utils/commit/3d4c9e1110ce0f9d7915685f1a43752bdda3e085))
* pre-solved environments for mulled tests ([#1082](https://github.com/bioconda/bioconda-utils/issues/1082)) ([af1ac9c](https://github.com/bioconda/bioconda-utils/commit/af1ac9c8ddfa17f4055aab065aa61b93973f8c30))


### Bug Fixes

* add raise_for_status to CircleCI workflow API call ([#1083](https://github.com/bioconda/bioconda-utils/issues/1083)) ([2f558ee](https://github.com/bioconda/bioconda-utils/commit/2f558ee0664af9fbd4a2435ca8d7001c6c28cb19))

## [4.0.0](https://github.com/bioconda/bioconda-utils/compare/v3.9.2...v4.0.0) (2026-02-11)


### ⚠ BREAKING CHANGES

* also find tests under outputs, ensure all outputs have tests, ensure outputs names are different from package name ([#1057](https://github.com/bioconda/bioconda-utils/issues/1057))

### Bug Fixes

* also find tests under outputs, ensure all outputs have tests, ensure outputs names are different from package name ([#1057](https://github.com/bioconda/bioconda-utils/issues/1057)) ([dd17aa7](https://github.com/bioconda/bioconda-utils/commit/dd17aa76b2410901649fb33c14e876452d26be1b))
* update actions ([#1076](https://github.com/bioconda/bioconda-utils/issues/1076)) ([e301b5c](https://github.com/bioconda/bioconda-utils/commit/e301b5c6d8139ce80fddb7262c9355cf3284320b))
* Update anaconda-client version to 1.14.* ([#1075](https://github.com/bioconda/bioconda-utils/issues/1075)) ([0bdd2a9](https://github.com/bioconda/bioconda-utils/commit/0bdd2a9202ec84e00c0ec923a24fafe68e9ee3a0)), closes [#1074](https://github.com/bioconda/bioconda-utils/issues/1074)

## [3.9.2](https://github.com/bioconda/bioconda-utils/compare/v3.9.1...v3.9.2) (2026-01-09)


Expand Down
1 change: 1 addition & 0 deletions bioconda_utils/artifacts.py
Original file line number Diff line number Diff line change
Expand Up @@ -216,6 +216,7 @@ def get_circleci_artifacts(check_run, platform):
# Use API v2 because v1.1 does not have a workflow endpoint
url_wf = f"https://circleci.com/api/v2/workflow/{circleci_workflow_id}/job"
res_wf = requests.get(url_wf, headers=headers)
res_wf.raise_for_status()
json_wf = json.loads(res_wf.text)

if len(json_wf["items"]) == 0:
Expand Down
2 changes: 1 addition & 1 deletion bioconda_utils/bioconda_utils-requirements.txt
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ boltons=25.0.* #
jsonschema=4.25.* # JSON schema verification
jinja2=3.1.* #

anaconda-client=1.13.* # anaconda_upload
anaconda-client=1.14.* # anaconda_upload
galaxy-tool-util=25.* # mulled test and container build
involucro=1.1.* # mulled test and container build
skopeo=1.15.* # docker upload
Expand Down
21 changes: 20 additions & 1 deletion bioconda_utils/build.py
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,8 @@ def build(
dag: Optional[nx.DiGraph] = None,
skiplist_leafs: bool = False,
live_logs: bool = True,
presolved_mulled_test: bool = True,
mulled_upload_target=None,
) -> BuildResult:
"""
Build a single recipe for a single env
Expand Down Expand Up @@ -198,6 +200,8 @@ def build(
if mulled_test:
logger.info("TEST START via mulled-build %s", recipe)
mulled_images = []
# Use pre-solved test env unless we need the mulled-build image for upload
use_presolved = presolved_mulled_test and not mulled_upload_target
for pkg_path in pkg_paths:
try:
report_resources(f"Starting mulled build for {pkg_path}")
Expand All @@ -206,6 +210,7 @@ def build(
base_image=base_image,
conda_image=mulled_conda_image,
live_logs=live_logs,
presolved=use_presolved,
)
except sp.CalledProcessError:
logger.error("TEST FAILED: %s", recipe)
Expand Down Expand Up @@ -366,6 +371,8 @@ def build_recipes(
live_logs: bool = True,
exclude: List[str] = None,
subdag_depth: int = None,
presolved_mulled_test: bool = True,
fast_resolve: bool = True,
):
"""
Build one or many bioconda packages.
Expand Down Expand Up @@ -482,7 +489,17 @@ def build_recipes(

logger.info("Determining expected packages for %s", recipe)
try:
pkg_paths = utils.get_package_paths(recipe, check_channels, force=force)
# When building with Docker, skip the expensive finalized render
# on the host since Docker's conda-build will re-solve anyway.
# Non-finalized metas use bypass_env_check which avoids costly
# dependency resolution. The --no-fast-resolve flag can override this.
finalize = (docker_builder is None) if fast_resolve else True
pkg_paths = utils.get_package_paths(
recipe,
check_channels,
force=force,
finalize=finalize,
)
except utils.DivergentBuildsError as exc:
logger.error(
"BUILD ERROR: packages with divergent build strings in repository "
Expand Down Expand Up @@ -521,6 +538,8 @@ def build_recipes(
record_build_failure=record_build_failures,
skiplist_leafs=skiplist_leafs,
live_logs=live_logs,
presolved_mulled_test=presolved_mulled_test,
mulled_upload_target=mulled_upload_target,
)

if not res.success:
Expand Down
16 changes: 16 additions & 0 deletions bioconda_utils/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -619,6 +619,18 @@ def do_lint(
action="store_true",
help="Disable live logging during the build process",
)
@arg(
"--no-presolved-mulled-test",
action="store_true",
help="Disable pre-solved mulled tests: always use mulled-build to solve and install "
"the test environment from scratch.",
)
@arg(
"--no-fast-resolve",
action="store_true",
help="Disable fast resolve: always run the full finalized conda solver on the host, "
"even when building with Docker. Useful for debugging build string mismatches.",
)
@arg("--exclude", nargs="+", help="Packages to exclude during this run")
@arg(
"--subdag-depth",
Expand Down Expand Up @@ -652,6 +664,8 @@ def build(
record_build_failures=False,
skiplist_leafs=False,
disable_live_logs=False,
no_presolved_mulled_test=False,
no_fast_resolve=False,
exclude=None,
subdag_depth=None,
):
Expand Down Expand Up @@ -726,6 +740,8 @@ def build(
live_logs=(not disable_live_logs),
exclude=exclude,
subdag_depth=subdag_depth,
presolved_mulled_test=not no_presolved_mulled_test,
fast_resolve=not no_fast_resolve,
)
exit(0 if success else 1)

Expand Down
53 changes: 46 additions & 7 deletions bioconda_utils/lint/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -246,8 +246,20 @@ def run(self, recipe: _recipe.Recipe, fix: bool = False) -> List[LintMessage]:
for num, src in enumerate(source):
self.check_source(src, f"source/{num}")

# Run depends checks
self.check_deps(recipe.get_deps_dict())
# Run depends checks, per outputs: package if necessary
outputs = recipe.get("outputs", dict())
deps = recipe.get_deps_dict()
if outputs:
for i in range(len(outputs)):
output_location = f"outputs/{i}/"
# filter down to dependencies for this outputs: package
output_deps = dict()
for dep in deps:
if any(output_location in d for d in deps[dep]):
output_deps[dep] = deps[dep]
self.check_deps(output_deps, output_location)
else:
self.check_deps(deps, "")

return self.messages

Expand All @@ -270,7 +282,7 @@ def check_source(self, source: Dict, section: str) -> None:
``source/0`` (1,2,3...).
"""

def check_deps(self, deps: Dict[str, List[str]]) -> None:
def check_deps(self, deps: Dict[str, List[str]], package_location: str) -> None:
"""Execute check on recipe dependencies

Example format for **deps**::
Expand All @@ -287,6 +299,12 @@ def check_deps(self, deps: Dict[str, List[str]]) -> None:
Args:
deps: Dictionary mapping requirements occurring in the recipe
to their locations within the recipe.
package_location: Path to the main location for the build and
requirements sections. Empty string for the top
level in single-package recipes, something like
``outputs/0/`` for recipes with packages specified
in an outputs section.

"""

def fix(self, message, data) -> LintMessage:
Expand Down Expand Up @@ -438,6 +456,17 @@ class jinja_render_failure(LintCheck):
"""


class skipping_recipe(LintCheck):
"""skipping linting of this recipe as requested

As specified via ``extra: skip-recipes:``, this recipe is being
skipped during linting. This is meant for the linter test
suite.
"""

severity: Severity = INFO


class unknown_check(LintCheck):
"""Something went wrong inside the linter

Expand Down Expand Up @@ -499,13 +528,11 @@ def __init__(
)
self.checks_dag = dag

def order_and_load_checks(self):
try:
self.checks_ordered = reversed(list(nx.topological_sort(dag)))
self.checks_ordered = reversed(list(nx.topological_sort(self.checks_dag)))
except nx.NetworkXUnfeasible:
raise RuntimeError("Cycle in LintCheck requirements!")
self.reload_checks()

def reload_checks(self):
self.check_instances = {str(check): check(self) for check in get_checks()}

def get_skiplist(self) -> Set[str]:
Expand Down Expand Up @@ -568,6 +595,7 @@ def lint(self, recipe_names: List[str], fix: bool = False) -> bool:

"""
for recipe_name in utils.tqdm(sorted(recipe_names)):
self.order_and_load_checks()
try:
msgs = self.lint_one(recipe_name, fix=fix)
except Exception:
Expand Down Expand Up @@ -597,6 +625,17 @@ def lint_one(self, recipe_name: str, fix: bool = False) -> List[LintMessage]:
check_cls = recipe_error_to_lint_check.get(exc.__class__, linter_failure)
return [check_cls.make_message(recipe=recipe, line=getattr(exc, "line"))]

# collect recipes to skip
if isinstance(recipe.get("extra/skip-recipes", []), list):
if any(
recipe_name.endswith(name) for name in recipe.get("extra/skip-recipes")
):
return [
skipping_recipe.make_message(
recipe=recipe,
)
]

# collect checks to skip
checks_to_skip = set(self.skip[recipe_name])
checks_to_skip.update(self.exclude)
Expand Down
32 changes: 23 additions & 9 deletions bioconda_utils/lint/check_build_help.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ class should_use_compilers(LintCheck):
"rust",
)

def check_deps(self, deps):
def check_deps(self, deps, _package_location):
for compiler in self.compilers:
for location in deps.get(compiler, []):
self.message(section=location)
Expand All @@ -55,7 +55,7 @@ class compilers_must_be_in_build(LintCheck):

"""

def check_deps(self, deps):
def check_deps(self, deps, _package_location):
for dep in deps:
if dep.startswith("compiler_"):
for location in deps[dep]:
Expand Down Expand Up @@ -101,7 +101,7 @@ def _check_line(line: str) -> bool:
return True
return False

def check_deps(self, deps):
def check_deps(self, deps, _package_location):
if "setuptools" not in deps:
return # no setuptools, no problem

Expand All @@ -127,7 +127,7 @@ class cython_must_be_in_host(LintCheck):
- cython
"""

def check_deps(self, deps):
def check_deps(self, deps, _package_location):
if "cython" in deps:
if any("host" not in location for location in deps["cython"]):
self.message()
Expand All @@ -146,13 +146,13 @@ class cython_needs_compiler(LintCheck):

severity = WARNING

def check_deps(self, deps):
def check_deps(self, deps, _package_location):
if "cython" in deps and "compiler_c" not in deps and "compiler_cxx" not in deps:
self.message()


class missing_run_exports(LintCheck):
"""Recipe should have a run_exports statement that ensures correct pinning in downstream packages
"""Recipe should have a run_exports statement for each package, ensuring correct pinning in downstream packages

This ensures that the package is automatically pinned to a compatible version if
it is used as a dependency in another recipe.
Expand All @@ -163,6 +163,15 @@ class missing_run_exports(LintCheck):
libraries) but also for e.g. Python packages, as those might also
introduce breaking changes in their APIs or command line interfaces.

A ``run_exports:`` specification should be specified in the relevant
``build:`` section for all packages that are built. This means either:
(i) in the main ``build`` section, if only one package is built from this
recipe, or:
(ii) in each outputs' ``build:`` section, if multiple ``outputs:`` are
specified. In this case, the main recipe ``build:`` section does not
refer to a package that is being built, but only to the recipe, so a
``run_exports:`` section for it does not make sense.

We distinguish between four cases.

**Case 1:** If the software follows semantic versioning (or it has at least a normal version string (like 1.2.3) and the actual strategy of the devs is unknown), add run_exports to the recipe like this::
Expand Down Expand Up @@ -210,6 +219,11 @@ class missing_run_exports(LintCheck):
"""

def check_recipe(self, recipe):
build = recipe.meta.get("build", dict())
if "run_exports" not in build:
self.message()
outputs = recipe.get("outputs", dict())
if outputs:
build_sections = [o.get("build", dict()) for o in outputs]
else:
build_sections = [recipe.meta.get("build", dict())]
for build in build_sections:
if "run_exports" not in build:
self.message()
Loading
Loading