fix!: also find tests under outputs, ensure all outputs have tests, ensure outputs names are different from package name

[dlaehnemann]

This pull request is motivated by trying to add a subpackage to an existing package and trying to get the syntax right:
bioconda/bioconda-recipes#58141

I think I have to add both the main and the subpackage under an outputs: section, and the linter then complains that it doesn't find any global test: section. So this PR makes the linter go through each subpackage if an outputs: section exists, and checks whether all of them have a valid test: section specified.

[dlaehnemann]

I can't ask for Reviewers on issues in this org, it seems. And I'm not sure whom to ping here for a review. So I'll just ping the top two contributors, @daler and @epruesse . Fell free to pass this on or let me know whom else to ping (or where).

[daler]

Thanks @dlaehnemann! Can you add a test case for this? test_lint.py is the runner; lint_cases.yaml has the test cases. You'll probably need to add a new recipe (say, three) to the setup section of the lint cases so you have something with outputs: to test.

[dlaehnemann]

Will do, thanks for the detailed pointers!

[dlaehnemann]

OK, good thing you asked for a test here, because obviously, it's complicated. The latest commit makes this work for test: being specified under outputs:, but generally the current outputs: behavior with conda-build is a bit of a mess. I gathered a bit of the messiness in this conda-build docs pull request:
conda/conda-build#5767

Long story short:

1. Never use the same name for the main level package name and a subpackage name under outputs:.
2. In the future, you can only have package outputs generated EITHER at the main level OR at the outputs: level. So once you specify outputs:, there is no build happening at the main level any more. The subpackages can then only share some of the info that then gets deep-merged into all of the subpackages. The plans are only implemented in rattler build so far, but are a generally accepted Conda Enhacement Proposal:
   https://conda.org/learn/ceps/cep-0014/#outputs-section

So I think the way forward for bioconda on specifying multiple outputs: could be to simply update all the recipes that currently use multiple outputs: to adhere to the format that CEP 14 suggests. And then adjust the linting to already check that recipes already conform to this future format. This format is then at least a bit easier to enforce, because it is explicit and has clearly defined requirements for implementations of it.

And are there any plans for bioconda to switch to rattler-build at some point? As mentioned above, that project has already implemented CEP14:
https://github.com/prefix-dev/rattler-build/blob/a40d91cacce345a40c8fa4ab6eddb99ecce77995/docs/reference/recipe_file.md#the-recipe-spec

And here's where the spec lives as a JSON schema:
https://github.com/prefix-dev/recipe-format

[dlaehnemann]

And maybe regarding the direct way forward with this pull request, I would like to know if there is agreement to work towards the future recipe.yaml format (while staying functional with the current underspecified meta.yaml format).

If so, I would adjust this pull request with a suggestion of a more general way to ensure compatibility with both via stricter linting. And I would then do a follow-up pull request to update all the recipes that already use outputs: in bioconda, which seem to be only 7 recipes at the time of writing:
https://github.com/search?q=repo%3Abioconda%2Fbioconda-recipes%20%22outputs%3A%22&type=code

[daler]

@dlaehnemann thanks for this investigation! Regarding plans to switch to rattler-build & recipe.yaml I don't think we have a formal plan or timeline , but I do think it's safe to say we'd move in that direction. So your proposal to support both and adjust the 7 existing accordingly sounds like a good one.

[dlaehnemann]

I just came across this older investigation of mine, and I think it is still valid and the changes needed. I updated the pull request according to the latest discussion here and this is now ready for review. @daler , would you have time to have a look?

[dlaehnemann]

Just pinging @daler for an update, here. Or maybe @aliciaaevans could also provide a review?

[aliciaaevans]

@dlaehnemann I'm taking a look, but need to dig through some of the links you provided to fully wrap my head around this. I guess the main concern is any implications updating the existing recipes that use outputs, which seems to be about 7 recipes. That's going to result in either a different recipe name or a different package name, right?

[dlaehnemann]

Thanks for having a look, and yes, that is what I would expect from the changes made here and the requirements of CEP14.

And looking at it, I just realized that one of the hits in the search is just an Outputs: in the description:, so it would be 6 recipes that I would have to update.

[dlaehnemann]

Just another ping @aliciaaevans , so this doesn't slip off the radar. And is there anything else I can do to help, to move this along?

[aliciaaevans]

Please try to get acknowledgement/approval from recipe maintainers when you do the updates. If there's an issue, since this is just a lint, it could be overridden in a recipe if there's a problem. @johanneskoester may have an insight since Snakemake is one of the packages using the outputs section. But I think as long as the end result packages in the channel stay the same, it's ok to update the recipe as needed. Looks like openms already had this done.

There is an existing issue I can see on https://bioconda.github.io/recipes/openms-meta/README.html that the plots showing downloads don't work because they are based on the recipe name. So that needs to be fixed at some point.

[dlaehnemann]

Yes, my plan was to do one PR per recipe and ping in the maintainers, as they should know best.

Good catch with the download stats plot. I'll have a look if I can also fix that while we are at this. I worry I'll otherwise forget soon enough.