Suggestion: Consider making Aarch64 builts ?

[DrYak]

A weird outlandish suggestion. Given that:

• Apple is building future laptops with their own homemade ARM CPUs (Mac Books - the most popular laptop among devs that aren't hardcore followers of the way of the Penguin :-D )
• A Fujistu's ARM 64-based super computer is in the top 500.
• Raspberry foundation has released a Pi board that has 8GB RAM (This begins to be quite descent to run some simple bioinformatics tasks, while being easy to be used by students, thanks to be being are cheap and ubiquitous)
• and other SBC manufacturer are even building clusters of such boards (making a possible environment to build and test Aarch64 code).
• Upstream Conda-Forge already has Linux Aarch64 builds.

...would the bioconda project eventually consider enabling Aarch64 builds of its package ?

[epruesse]

Eventually? Yes. Today? No.

You are welcome to get the necessary infrastructure in place to support that kind of build, we'd welcome a PR. Many packages require extra effort for every build target though, and unless there is a real need, it's difficult to justify. Just MacOS in addition to Linux already is a burden.

[mr-c]

@epruesse Thanks for the response. What's the process to add arm64 / aarch64 support to an existing recipe? We could document that, for users who want to self-service.

[epruesse]

We can't just add it for a single recipe. You'd have to start by creating the infrastructure parts for the new build target. Then you'd try to build all 6000 recipes. Many will fail, so you will have to fix them, making sure that they still build under both other target platforms. Once you have a reasonable set of packages done, say 5000 or so, you could give up and mark the rest with skip: True # [aarch64]. But you will be touching thousands of packages to get this on the road.

[JeffUnderhill]

@DrYak Your original post also missed that Arm-based servers are also now readily available in the AWS cloud with Graviton2. I can help with access to native Arm64 build infrastructure if that will help someone work on building/testing of Arm64? As a first pass it would be useful to get a view of how many recipes may just work and then analyze the characteristics of those that fail to see if there are any patterns to the failure modes that would help address 'blocks' of failures.

[epruesse]

For official packages, we use free public services only. Initially it was TravisCI, now CircleCI and Github Actions for the bulk bioconductor packages. Conda-forge is using Azure. I assume that they are using a cross compiler. Personally I think we have a lot of other areas that need work, but if you guys want to work on this, as I said, you are welcome. Head over to bioconda-utils, make a fork, set up some build chain. If in doubt you can always post the packages to a custom conda channel to start with.

[bgruening]

As a first pass it would be useful to get a view of how many recipes may just work and then analyze the characteristics of those that fail to see if there are any patterns to the failure modes that would help address 'blocks' of failures.

That is true. @JeffUnderhill you could for example try to rebuild bioconda on those AWS instances and see how far you will come.

[mjsteinbaugh]

Circling back to this, I'm up for volunteering some of my time to work on aarch64 (ARM) builds.

[tfenne]

Seems like this is also under discussion in bioconda/bioconda-utils#706 with more recent comments.

[jeffreykstone]

Yes please!

[corneliusroemer]

Notably, conda-forge has an osx64-arm channel - so it would be great if this could get off the ground for bioconda, too.

At some point in the near future, almost all users of osx will be using apple silicon - at which point it would be weird to use emulation for all of bioconda.

Are there free-to-use resources now to build for osx-ARM? Things may have changed a lot since the first comments ~2y ago.

[colinbrislawn]

Are there free-to-use resources now to build for osx-ARM?

osx-arm64 support is on the roadmap for GitHub actions! More discussion here.

[corneliusroemer]

Per Github roadmap, osx-arm64 support is now scheduled to arrive in Q3 of 2023 🎉

However, this wouldn't help with Linux aarch64. Is there a reason that we need to use Github actions? Isn't the current CI Azure based anyways?

[Yikun]

Just curious, how many build resources are needed to support bioconda linux aarch64?

Before the Github-hosted host supported linux aarch64, the self hosted runner or run-on-arch-action (qemu based) Could it be an alternative as aarch64 build resource?

[martin-g]

I use the following script to build Bioconda packages on Linux ARM64:

#!/usr/bin/env bash

# a Conda virtual environment to keep the things clean and tidy
conda activate biotest

YEAR=$(date +%Y)
MONTH=$(date +%m)
DAY=$(date +%d)
HOUR=$(date +%H)
MINUTE=$(date +%M)

LOG_FOLDER=logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}
mkdir -p ${LOG_FOLDER}

BIOCONDA_RECIPES_ROOT=${1:-$PWD/recipes}
DOCKER_BASE_IMAGE="ghcr.io/yikun/bioconda-utils-build-env-cos7-aarch64"
MULLED_CONDA_IMAGE="ghcr.io/martin-g/create-env-aarch64"

for recipe in $(ls ${BIOCONDA_RECIPES_ROOT}); do
	echo -e "\n\nGoing to build recipe '$recipe'";
	
	bioconda-utils build \
		--docker \
		--mulled-test \
		--docker-base-image ${DOCKER_BASE_IMAGE} \
		--mulled-conda-image ${MULLED_CONDA_IMAGE} \
		--packages ${recipe} \
	        2>&1 | tee ${LOG_FOLDER}/${recipe}.log
done

So far I have built 100+ recipes. Some of them fail, some pass, but the tooling (bioconda-utils) seems to work just fine!

[phiweger]

Entering Q4 of 2023 -- any update on this issue?

[martin-g]

Entering Q4 of 2023 -- any update on this issue?

• feat: add initial support for arm builds #40550
• Make base images multiarch bioconda-containers#55 and Add aarch64 base images, update to Debian 12.2 bioconda-containers#58

[phiweger]

Entering Q4 of 2023 -- any update on this issue?

• feat: add initial support for arm builds #40550
• Make base images multiarch bioconda-containers#55 and Add aarch64 base images, update to Debian 12.2 bioconda-containers#58

so nearly there, awesome.

[pditommaso]

Can't wait for this!

[martin-g]

Here is the updated version of the script I shared earlier. Now it uses the official Docker images:

#!/usr/bin/env bash

conda activate biotest

YEAR=$(date +%Y)
MONTH=$(date +%m)
DAY=$(date +%d)
HOUR=$(date +%H)
MINUTE=$(date +%M)

LOG_FOLDER=logs/${YEAR}/${MONTH}/${DAY}/${HOUR}/${MINUTE}
mkdir -p ${LOG_FOLDER}

BIOCONDA_RECIPES_ROOT=${1:-$PWD/recipes}
RECIPES=$2
if [ -z $RECIPES ]; then
        RECIPES=$(ls ${BIOCONDA_RECIPES_ROOT})
fi
echo "Recipes: $RECIPES"

# New official Docker images that support Linux ARM64 !
DOCKER_BASE_IMAGE="quay.io/bioconda/bioconda-utils-build-env-cos7-aarch64"
MULLED_CONDA_IMAGE="quay.io/bioconda/create-env"

for recipe in $RECIPES; do

                echo -e "\n\nGoing to build recipe '$recipe'";
                
                bioconda-utils build \
                        --docker \
                        --mulled-test \
                        --force \
                        --docker-base-image ${DOCKER_BASE_IMAGE} \
                        --mulled-conda-image ${MULLED_CONDA_IMAGE} \
                        --packages ${recipe} \
                        2>&1 | tee ${LOG_FOLDER}/${recipe}.log
done

[corneliusroemer]

There's a lot of progress! Looks like aarch64 will happen in the next few weeks. And eventually also arm64. See:

• ARM Documentation updates bioconda-docs#16

[pditommaso]

Can't wait for it to happen!

[rob-p]

Is this supported yet?

[emiliofernandes]

I see there is a lot of progress in the last weeks!
But it is not very clear to me whether there are any stoppers at the moment.
I'd be glad to help with testing and/or coding if such help is needed !

[Adamtaranto]

@martin-g Does this go in the build.sh script in a Bioconda recipe?

Here is the updated version of the script I shared earlier. Now it uses the official Docker images:
...

[martin-g]

@Adamtaranto No! This is a custom shell script that I use locally to build/test recipes on Linux ARM64.
Once Bioconda officially adds the support for Linux ARM64 then you will just need to add to following to a recipe:

extra:
  additional-platforms:
    - linux-aarch64

See bioconda/bioconda-utils#923

[corneliusroemer]

Per the docs, linux aarch64 now appears to be supported.

https://github.com/bioconda/bioconda-docs/pull/16/files

[erikalmecdc]

There is no support for linux-aarch64 for sra-tools, a vital package for any bioinformatician. Will these be added in the near future?

[martin-g]

@erikalmecdc sra-tools depends on many perl-** packages which need to provide linux-aarch64
builds ...
#46237

[jmarshall]

As per #33333 (comment), the infrastructure is now in place for aarch64 builds for both Linux and macOS.