Build env as others

[daler]

In #69, the GH actions workflow yaml was ported over from bioconda-utils, and thanks to @martin-g's efforts, refactored to support arm64 and manifests.

Upon thinking about this some more, the ported-over yaml was written quite differently, since it was living in a different repo with nothing to really compare to. But now it's living next to yamls that largely use the same steps. In the interest of long-term maintenance and sustainability, I figured it would be best to have everything built consistently here rather than have the build-env be the odd one out (and requiring understanding a different way of creating the same final outcome).

I finally sat down to understand the existing commands by @mbargull, and added some comments here to help the next reader. Then used that newfound knowledge to rewrite the build-env image building to match everything else here.

The end result is intended the same way as the others in this repo, with multi-arch images and manifest all pushed to quay.io.

Changes required to make that happen included using build args to specify the base container (which was a little bit awkward since conda-forge and quay.io call different archs by different names; see the arch_and_image for-loop vars), and a Dockerfile.test just to make sure bioconda-utils is installed correctly.

[daler]

While tests are passing, and we're in better shape for arm containers, this is still broken as a whole because create-env's dockerfile expects a build-env container with the same version in quay.io from which to copy out requirements. In other words, the create-env and the build-env are really tightly coupled to each other and the previous way of building containers.

As-is, a new create-env container will not be able to be built.

A not-great solution would be to tag the bioconda-utils-build-env-cos7 container with the bioconda-utils version instead of the hard-coded major/minor. But the problem is that it would introduce a race condition between create-env and build-env dockerfiles. That is, within this repo we would need to require build-env to complete first and upload an image to quay.io and only then allow create-env to pull that tag. I think it worked OK before when the build-env container was built in another repo because you could manually trigger that and then later manually trigger this repo to make the create-env container.

But nowadays, we're already factoring out the bioconda-utils version info into bioconda-common/common.sh, and we now have the uniform build script over in bioconda-common.

So I think the better approach is to refactor the create-env and build-env Dockerfiles to use the exact same method of installing bioconda-utils as e.g. https://github.com/bioconda/bioconda-recipes/blob/master/azure-pipeline.yml.

Even create-env uses a different way of installing conda! When overhauling bioconda-common, I had focused on unifying how it was installed on CI/CD (which has been working nicely I think) but never made it to overhauling the containers.

All that said, I am unconvinced that nowadays we actually need bioconda-utils inside the containers! The build script used by the container only needs conda. It should probably have the requirements for bioconda-utils (to be careful about things like the version of conda-build being used), but I don't think bioconda-utils itself needs to be installed.

[martin-g]

create-env workflow can depend on build-env via the workflow_run event - https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#workflow_run:
e.g.

on:
  workflow_run:
    workflows: [Build & Push: bioconda-utils-build-env-cos7]
    types:
      - completed

But if bioconda-utils is not needed in the containers then I guess some code could be removed instead of adding new.

[daler]

Yeah, I'm trying to figure out if we can get away with building packages just with the create-env image and do away with bioconda-util-build-env-cos7 altogether. That will simplify things a little bit.

However, #73, which simulates merging to main and theoretically should support pushing the images to quay.io, appears broken (it's not authenticating). Trying to figure out how to debug that.

[martin-g]

Same as #68 (comment)
Moving the registry name and the username from secrets to vars will make it more clear what it is attempting to do.
Maybe #68 (comment) is also related.

But the same secrets work fine for the other workflows, so it must be something else...

[daler]

https://github.com/redhat-actions/push-to-registry seems to suggest that quay.io/bioconda would be OK, but will check with just typing that in...I moved over to #73 and implemented an image prefix for better testing. My intention is that once that's working, I'll change from targeting test-push to target main and merge what works there into this PR.

[daler]

Reporting back on some findings....

1. It turns out quay.io requires manual intervention. You must push an image first so the repo exists (quay.io/bioconda/<image> is a repo in this context). Then you give the bot user permissions to that repo. Might be able to avoid that by giving the bot admin access but I don't think we want to go there.

2. I realized we do in fact need the build-env container and the conda-env container -- the former needs conda-build to actually build the package; the latter should not have conda-build so as to avoid having undeclared dependencies installed.

3. The current create-env container extracts only the actually-installed conda version from the build-env container. This is still important to do, so the dependency of jobs should reflect this. I'm currently trying to figure out what version is best to use -- a global major.minor, or tie the build-env to bioconda-utils?

[daler]

This is all working now in bioconda/bioconda-utils#953, so closing this.