feat: uploading linux-aarch64 biocontainers

[wjunLu]

What do I want to do?

The bioconda/biocomda-recipes repository currently supports the build and distribution of .conda-formatted packages for multiple architectures, including linux-x64 and linux-aarch64.

Nevertheless, it only releases biocontainers for the linux-x64 architecture. With the growing number of developers adopting ARM environments for their work, biocontainers of inux-aarch64 should also be made available on Quay to serve users' needs.

How bioconda/biocomda-recipesbuild linux-aarch64 biocontainers?

In build-and-test stage of bioconda/biocomda-recipes CI, we can see that it does build linux-aarch64 biocontainers:

bioconda-utils build recipes config.yml \
                  --lint --docker --mulled-test \
                  --docker-base-image "quay.io/bioconda/bioconda-utils-build-env-cos7-$(arch):${BIOCONDA_UTILS_TAG#v}" \
                  --git-range origin/master HEAD

where --mulled-test is set to build linux-aarch64 biocontainers, and this can be found from circleci artifacts:
(see https://app.circleci.com/pipelines/github/bioconda/bioconda-recipes/165751/workflows/85460b3f-0f6b-4ba9-b8bb-9c449b19ec07/jobs/301957?utm_campaign=vcs-integration-link&utm_medium=referral&utm_source=github-checks-link&utm_content=summary&invite=true#step-109-1902_77):

How bioconda/biocomda-recipes release biocontainers currently?

As we can see the bioconda/biocomda-recipes repository only releases linux-x64 biocontainers:

• linux-x64 merge workflow:

bioconda-utils handle-merged-pr recipes config.yml \
            --repo bioconda/bioconda-recipes \
            --git-range ${GITHUB_SHA}~1 ${GITHUB_SHA} \
            --quay-upload-target biocontainers \
            --fallback build \
            --artifact-source github-actions

where --quay-upload-target biocontainers is used for pushing linux-64 biocontainers to quay.io.

• linux-aarch64 merge workflow:

bioconda-utils handle-merged-pr recipes config.yml \
              --repo bioconda/bioconda-recipes \
              --git-range ${CIRCLE_SHA1}~1 ${CIRCLE_SHA1} \
              --fallback build \
              --artifact-source circleci

Since there is no --quay-upload-target used, linux-aarch64 biocontainers will not be pushed.

What is changed?

Only one point needs modify in bioconda-utils:

if mulled_upload_target:
                    quay_login = os.environ['QUAY_LOGIN']

                    pattern = f"{tmpdir}/*/images/*.tar.gz"
                    if artifact_source == "circleci":
                        pattern = f"{tmpdir}/*/*/images/*.tar.gz"

where if artifact_source == "circleci": pattern = f"{tmpdir}/*/*/images/*.tar.gz" is added.

This is due to differences in the biocontainer image's storage paths for linux-aarch64, for example
If biocontainer artitfact is: https://output.circle-artifacts.com/output/job/7643a234-bf61-412e-ac87-69875fe1f8f1/artifacts/0/tmp/artifacts/images/swarm:3.1.5--h163da20_3.tar.gz.

After using the following code to store above biocontainer:

elif artifact_source == "circleci":
                    artifact_dir = os.path.join(tmpdir, *(artifact.split("/")[-4:-1]))
                    artifact_path = os.path.join(tmpdir, artifact_dir, os.path.basename(artifact))
                    Path(artifact_dir).mkdir(parents=True, exist_ok=True)
                    download_artifact(artifact, artifact_path, artifact_source)

This final local stored path of biocontainer is: {tmpdir}/tmp/artifacts/images/swarm:3.1.5--h163da20_3.tar.gz. Compared to the current pattern, there is an additional level of tmp/ directory.

Local Tests

The result looks good, and the biocontainer has been successfully pushed to my quay account

[wjunLu]

Hi @ALL!
Could anyone help see this?

[bgruening]

That might be interesting to @daler

[johanneskoester]

Seems reasonable to me to upload the aarch containers as well. However, I wonder why the directory structure is different. Maybe we can instead just unify the location where things are stored between circleci and the other CIs?

[johanneskoester]

Maybe this is fixed by #1051?