feat: containers overhaul

[daler]

Previously, the various bioconda images were built in different places (bioconda-containers and bioconda-utils) using different methodologies and repeated-but-not-quite-the-same code.

This PR aims to consolidate all image building into a single location

The location is here in bioconda-utils because 1) images are tightly coupled to the bioconda-utils version, 2) it is much more straightforward to handle everything in the same GitHub Actions workflow rather than try to coordinate across repos, and 3) consolidating means we can actually capture the dependencies between images (in ways that were not actually being captured previously).

Here are the dependencies, for reference:

graph TD;
    base-busybox-->build-env;
    bioconda-utils-->build-env;
    build-env-->create-env;
    create-env-->bot;
    create-env-->A["final builds"];
    build-env-->A["final builds"];
    base-debian-->A["final builds"];
    bioconda-utils-->A["final builds"];
    style bioconda-utils fill:#f9f,stroke:#333,stroke-width:4px

Loading

See dockerfile inventory for some background on how the images are used. There are details on that page that will need to be updated once this is merged in and the containers are being used in production.

Briefly, this PR:

• collects Dockerfiles from all relevant images, putting each into a directory in images
• adds utility infrastructure like images/image_config.sh and images/build.sh to control the builds
• adds a new .github/workflows/build-images.yml GHA workflow to orchestrate the building
• removes earlier infrastructure for building the create-env image
• makes some changes in bioconda-utils to support more flexible environment variable configuration

See images/README.md for details, including how to run locally.

This replaces #1046, #959, and #953 which had a LOT of troubleshooting; commits have been cleaned up and rebased into this this PR. Those other PRs attempted to encode the DAG in the GitHub Actions workflow and use caching in various ways for parallel builds. The problem was that it was too hard to keep all the moving parts in your head, especially returning to it after a while. I thought this complexity posed a risk to long-term maintenance burden. So this PR has all images created in the same GitHub Actions job, no clever caching or dependencies across GHA jobs. This has a time cost -- it takes about an hour to build everything -- but in practice these will not need to be rebuilt often.

The following is the proposed workflow for images, from PR through release-please. It includes thinking about which tags should get pushed when.

For example, a merge to master pushesmaster tagged manifests to quay.io. These are overwritten on each push to master. In this way, the :master tag sort of behaves like the latest tag conventionally does, but it only applies to the master branch. Note that we can't use the new release version number at this time because release-please has not been run (where it creates a new tag and release).

Then when release-please runs, it takes whatever existing manifest is tagged as master and creates two new tags: 1) a tag reflecting the bioconda-utils release and 2) latest.

So master will always refer to the latest images built on the master branch, and latest will always refer to the most recent release.

flowchart TD
a[Submit PR]-->|image building disabled?|b[test with *latest* quay.io images]
b-->m[merge to master]
a-->|default: image building enabled|c[build images locally]
c-->d[test with local images]
d-->m
m-->f[build images locally]
f-->g[test with local images]
g-->h[push manifests with *master* tag to quay.io]
h-->i[release please]
i-->j[tag *master* manifests as *latest* on quay.io]
i-->k[tag *master* manifests as *release version* on quay.io]
style i fill:#f9f,stroke:#333,stroke-width:4px
style m fill:#f9f,stroke:#333,stroke-width:4px
style a fill:#f9f,stroke:#333,stroke-width:4px

Loading

[wjunLu]

@ALL!
May I ask when this PR will be merged? Our team are also interested in ARM biocontainers.

bioconda/bioconda-recipes#58943

[sachalau]

Hello, can I ask where does the overhaul of the container building stand at the moment?

[bgruening]

We will start working on this after we have typed and upgraded the code base.

#1105

[sachalau]

Thanks a lot for the heads up !