fix: also find tests under outputs, ensure all outputs have tests

[dlaehnemann]

This pull request is motivated by trying to add a subpackage to an existing package and trying to get the syntax right:
bioconda/bioconda-recipes#58141

I think I have to add both the main and the subpackage under an outputs: section, and the linter then complains that it doesn't find any global test: section. So this PR makes the linter go through each subpackage if an outputs: section exists, and checks whether all of them have a valid test: section specified.

[dlaehnemann]

I can't ask for Reviewers on issues in this org, it seems. And I'm not sure whom to ping here for a review. So I'll just ping the top two contributors, @daler and @epruesse . Fell free to pass this on or let me know whom else to ping (or where).

[daler]

Thanks @dlaehnemann! Can you add a test case for this? test_lint.py is the runner; lint_cases.yaml has the test cases. You'll probably need to add a new recipe (say, three) to the setup section of the lint cases so you have something with outputs: to test.

[dlaehnemann]

Will do, thanks for the detailed pointers!

[dlaehnemann]

OK, good thing you asked for a test here, because obviously, it's complicated. The latest commit makes this work for test: being specified under outputs:, but generally the current outputs: behavior with conda-build is a bit of a mess. I gathered a bit of the messiness in this conda-build docs pull request:
conda/conda-build#5767

Long story short:

1. Never use the same name for the main level package name and a subpackage name under outputs:.
2. In the future, you can only have package outputs generated EITHER at the main level OR at the outputs: level. So once you specify outputs:, there is no build happening at the main level any more. The subpackages can then only share some of the info that then gets deep-merged into all of the subpackages. The plans are only implemented in rattler build so far, but are a generally accepted Conda Enhacement Proposal:
   https://conda.org/learn/ceps/cep-0014/#outputs-section

So I think the way forward for bioconda on specifying multiple outputs: could be to simply update all the recipes that currently use multiple outputs: to adhere to the format that CEP 14 suggests. And then adjust the linting to already check that recipes already conform to this future format. This format is then at least a bit easier to enforce, because it is explicit and has clearly defined requirements for implementations of it.

And are there any plans for bioconda to switch to rattler-build at some point? As mentioned above, that project has already implemented CEP14:
https://github.com/prefix-dev/rattler-build/blob/a40d91cacce345a40c8fa4ab6eddb99ecce77995/docs/reference/recipe_file.md#the-recipe-spec

And here's where the spec lives as a JSON schema:
https://github.com/prefix-dev/recipe-format

[dlaehnemann]

And maybe regarding the direct way forward with this pull request, I would like to know if there is agreement to work towards the future recipe.yaml format (while staying functional with the current underspecified meta.yaml format).

If so, I would adjust this pull request with a suggestion of a more general way to ensure compatibility with both via stricter linting. And I would then do a follow-up pull request to update all the recipes that already use outputs: in bioconda, which seem to be only 7 recipes at the time of writing:
https://github.com/search?q=repo%3Abioconda%2Fbioconda-recipes%20%22outputs%3A%22&type=code

[daler]

@dlaehnemann thanks for this investigation! Regarding plans to switch to rattler-build & recipe.yaml I don't think we have a formal plan or timeline , but I do think it's safe to say we'd move in that direction. So your proposal to support both and adjust the 7 existing accordingly sounds like a good one.