Running arm32/v7 containers on an aarch64-only host

[Silex]

Hello,

Coming here from multiarch/qemu-user-static#77, it looks like qus works for i386, x84_64 but not arm:

gitlab-runner@docker-emacs:~$ uname -a
Linux docker-emacs 4.15.0-72-generic #81-Ubuntu SMP Tue Nov 26 12:21:09 UTC 2019 aarch64 aarch64 aarch64 GNU/Linux

gitlab-runner@docker-emacs:~$ docker run --rm --privileged aptman/qus -- -r
cat ./qemu-binfmt-conf.sh | sh -s -- --path=/qus/bin -r

gitlab-runner@docker-emacs:~$ docker run --rm --privileged aptman/qus -s -- -p
cat ./qemu-binfmt-conf.sh | sh -s -- --path=/qus/bin -p --suffix -static
Setting /qus/bin/qemu-i386-static as binfmt interpreter for i386
Setting /qus/bin/qemu-i386-static as binfmt interpreter for i486
Setting /qus/bin/qemu-alpha-static as binfmt interpreter for alpha
Setting /qus/bin/qemu-armeb-static as binfmt interpreter for armeb
Setting /qus/bin/qemu-sparc-static as binfmt interpreter for sparc
Setting /qus/bin/qemu-sparc32plus-static as binfmt interpreter for sparc32plus
Setting /qus/bin/qemu-sparc64-static as binfmt interpreter for sparc64
Setting /qus/bin/qemu-ppc-static as binfmt interpreter for ppc
Setting /qus/bin/qemu-ppc64-static as binfmt interpreter for ppc64
Setting /qus/bin/qemu-ppc64le-static as binfmt interpreter for ppc64le
Setting /qus/bin/qemu-m68k-static as binfmt interpreter for m68k
Setting /qus/bin/qemu-mips-static as binfmt interpreter for mips
Setting /qus/bin/qemu-mipsel-static as binfmt interpreter for mipsel
Setting /qus/bin/qemu-mipsn32-static as binfmt interpreter for mipsn32
Setting /qus/bin/qemu-mipsn32el-static as binfmt interpreter for mipsn32el
Setting /qus/bin/qemu-mips64-static as binfmt interpreter for mips64
Setting /qus/bin/qemu-mips64el-static as binfmt interpreter for mips64el
Setting /qus/bin/qemu-sh4-static as binfmt interpreter for sh4
Setting /qus/bin/qemu-sh4eb-static as binfmt interpreter for sh4eb
Setting /qus/bin/qemu-s390x-static as binfmt interpreter for s390x
Setting /qus/bin/qemu-aarch64_be-static as binfmt interpreter for aarch64_be
Setting /qus/bin/qemu-hppa-static as binfmt interpreter for hppa
Setting /qus/bin/qemu-riscv32-static as binfmt interpreter for riscv32
Setting /qus/bin/qemu-riscv64-static as binfmt interpreter for riscv64
Setting /qus/bin/qemu-xtensa-static as binfmt interpreter for xtensa
Setting /qus/bin/qemu-xtensaeb-static as binfmt interpreter for xtensaeb
Setting /qus/bin/qemu-microblaze-static as binfmt interpreter for microblaze
Setting /qus/bin/qemu-microblazeel-static as binfmt interpreter for microblazeel
Setting /qus/bin/qemu-or1k-static as binfmt interpreter for or1k
Setting /qus/bin/qemu-x86_64-static as binfmt interpreter for x86_64

gitlab-runner@docker-emacs:~$ docker run --rm -it i386/ubuntu bash -c 'uname -m'
i686

gitlab-runner@docker-emacs:~$ docker run --rm -it amd64/ubuntu bash -c 'uname -m'
x86_64

gitlab-runner@docker-emacs:~$ docker run --rm -it arm32v7/ubuntu bash -c 'uname -m'
standard_init_linux.go:211: exec user process caused "exec format error"
failed to resize tty, using default size

Any idea of what I should do? On https://askubuntu.com/questions/1090351/can-i-run-an-arm32-bit-app-on-an-arm64bit-platform-which-is-running-ubuntu-16-04 they suggest adding armhf as a foreign architecure but it looks random, I prefer to ask here first.

[umarcor]

Hi @Silex! It seems that you are trying to use qus on an aarch64 host to target armhf. I believe I explicitly excluded armhf/armv7 targets for aarch64 hosts because AFAIK all aarch64 devices are indeed armv8 which is capable of running either aarch64 or aarch32. The later is suppossed to be a superset of armhf/armv7.

Hence, in your case, I would expect arm32v7/ubuntu to work without qus. In fact, I have used either arm32v7 or arm64v8 on RPi 3 with raspbian (which is a 32 bit OS and should not be able to execute arm64v8), without qus.

May I know which device are you trying to run arm32v7 containers on?

Any idea of what I should do? On askubuntu.com/questions/1090351/can-i-run-an-arm32-bit-app-on-an-arm64bit-platform-which-is-running-ubuntu-16-04 they suggest adding armhf as a foreign architecure but it looks random, I prefer to ask here first.

In any case, I think you should not need to add libraries for a foreign architecture to your host. That is precisely the purpose of using a container. So, when using arm32v7 or arm64v8 you are effectively changing the set of standard libraries that the target application will find in the environment.

[Silex]

Hi @Silex! It seems that you are trying to use qus on an aarch64 host to target armhf. I believe I explicitly excluded armhf/armv7 targets for aarch64 hosts because AFAIK all aarch64 devices are indeed armv8 which is capable of running either aarch64 or aarch32. The later is suppossed to be a superset of armhf/armv7.
Hence, in your case, I would expect arm32v7/ubuntu to work without qus. In fact, I have used either arm32v7 or arm64v8 on RPi 3 with raspbian (which is a 32 bit OS and should not be able to execute arm64v8), without qus.

I thought so too before running into this issue 😅

May I know which device are you trying to run arm32v7 containers on?

From what I know it's a 96 core ThunderX system running on ubuntu 18.04. Feel free to ask me for more informations, here's parts of /proc/cpuinfo:

root@docker-emacs:~# cat /proc/cpuinfo 
processor	: 0
BogoMIPS	: 200.00
Features	: fp asimd evtstrm aes pmull sha1 sha2 crc32 cpuid
CPU implementer	: 0x43
CPU architecture: 8
CPU variant	: 0x1
CPU part	: 0x0a1
CPU revision	: 1

Any idea of what I should do? On askubuntu.com/questions/1090351/can-i-run-an-arm32-bit-app-on-an-arm64bit-platform-which-is-running-ubuntu-16-04 they suggest adding armhf as a foreign architecure but it looks random, I prefer to ask here first.

In any case, I think you should not need to add libraries for a foreign architecture to your host. That is precisely the purpose of using a container. So, when using arm32v7 or arm64v8 you are effectively changing the set of standard libraries that the target application will find in the environment.

Alright. If you have any suggestions 😉 Or questions I should ask the owner of this machine...

[Silex]

I just tested and confirm that without qus, only arm64v8/ubuntu works and all of i386/ubuntu, amd64/ubuntu and arm32v7/ubuntu fail.

[umarcor]

From what I know it's a 96 core ThunderX system running on ubuntu 18.04.

Alright. If you have any suggestions 😉 Or questions I should ask the owner of this machine...

I'm going to assume it's a recent version of ThunderX with support for 64bit only. I'd be glad if you could ask the owner about it. However, this is just curiosity (see below).

I just tested and confirm that without qus, only arm64v8/ubuntu works and all of i386/ubuntu, amd64/ubuntu and arm32v7/ubuntu fail.

That's partially good news. It means that qus works as expected.

• The qemu-user binaries that are included in the default aptman/qus image that is used on aarch64 host are extracted from http://ftp.debian.org/debian/pool/main/q/qemu/qemu-user-static_4.2-6_arm64.deb.
• As you see, qemu-arm-static, qemu-armeb-static, qemu-aarch64-static and qemu-aarch64-static exist; but there is no qemu-armhf-static. That feels like a packaging error... It doesn't make sense to have qemu-aarch64* packages in the DEB for an aarch64 host. However:
  • On the one hand, your logs shows that the *be versions are registered only, which is good.
  • On the other hand, it seems that qemu-armhf-static did never exist...

As a result, I'd say that you'd need qemu-arm-static to be registered, but for some reason it is not. Only aarch64_be and armbe are being registered. This leads me to think that it might be some issue with https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh. Et voilà: https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh#L139-L170

Since qemu-binfmt-conf.sh registers the targets by families, and arm/armhf/aarch64 is considered the same family, arm is not registered on aarch64: https://github.com/qemu/qemu/blob/master/scripts/qemu-binfmt-conf.sh#L314.

Hence, we have isolated the issue, I think. Now, I would suggest you to:

• Download the DEB package above, extract it and pick qemu-arm-static only.
• Build/copy some binary which is for armhf and execute ./qemu-arm-static yourbinary. Try ./yourbinary too. The former should work, the later should fail.

If that works as expected, we need to find the equivalent to running qemu-binfmt-conf.sh for your case only, in order to register qemu-arm-static with -p and have it loaded into memory. Doing so should allow you to run arm32v7 containers, and it should allow me to "fix" the fork of the script that we use here.

Unfortunately, playing with this can be risky. If we modify the script improperly and qemu-aarch-static gets registered on an aarch64 host, weird things will happen. That's why I suggest we go step by step and introduce commands manually first. Slow but secure.

[Silex]

@umarcor: thanks for the fantastic reply. I'll try your steps and report.
@vielmetti: can you answer about the hardware? is the box a ThunderX with support for 64bit only? The problem I have right now is that it does not seem to run 32 bits binaries, but I'll run more tests.

[vielmetti]

@Silex

ThunderX is 64 bit only, as is ThunderX2
Ampere eMag is 32 and 64 bit.

[Silex]

@vielmetti: thanks for confirming. Will try umarcor's idea which should work, will keep you posted.

[Silex]

@umarcor: I confirm your suggestion works:

root@docker-emacs:~# file hello
hello: ELF 32-bit LSB executable, ARM, EABI5 version 1 (GNU/Linux), statically linked, for GNU/Linux 3.2.0, BuildID[sha1]=99367b08573e864abd79831673e2fbbd7bb1f82f, not stripped

root@docker-emacs:~# ./hello
-bash: ./hello: cannot execute binary file: Exec format error

root@docker-emacs:~# ./qemu-arm-static hello
hello world

I cross compiled a basic hello.c using arm-linux-gnueabihf-gcc -static inside an arm64v8 container and copied the binary to the host. I extracted qemu-arm-static from apt download qemu-user-static, do you need me to test with your debian package instead? This box runs ubuntu 18.04.

[umarcor]

@vielmetti, thanks for clarifying.

@Silex, your tests are ok. The qemu-arm-static you extracted is likely to be the exact same version (or very similar), just retrieved with a different procedure.

I wrote a temporal fix for qemu-binfmt-conf.sh which splits family 'arm' into 'arm' and 'aarch64', and 'armeb' into 'armeb' and 'armeb64'. As a result, you should get all but aarch64 registered. In order to try it, please follow these steps:

• Download https://github.com/umarcor/qemu/blob/fix-aarch64only/scripts/qemu-binfmt-conf.sh to your host.
• Test that you can overwrite the copy inside the container with it:

$ docker run --rm --privileged -itv $(pwd)/qemu-binfmt-conf.sh:/qus/qemu-binfmt-conf.sh --entrypoint=sh aptman/qus
# cat /qus/qemu-binfmt-conf.sh | grep armeb64

grep should find some result.

• If successful, use it to run the "regular" command:

docker run --rm --privileged -v $(pwd)/qemu-binfmt-conf.sh:/qus/qemu-binfmt-conf.sh aptman/qus -s -- -p

That will hopefully allow you to run ./hello and to use arm32v7 containers on an aarch64 only host. It is not a clean solution, though, because it breaks backwards compatibility. Ideally, the behaviour you need would need to be explicitly enabled through some option/swtich. However, I'd like to think about it after you have tried the current proposal.

[Silex]

@umarcor: it works! 🎉

The only weird thing is that the first time I run it for other architectures it seems to output the output twice, but maybe that's just this particular session:

root@docker-emacs:~# docker run --rm -it i386/ubuntu bash -c 'uname -m'
i686i686

root@docker-emacs:~# docker run --rm -it i386/ubuntu bash -c 'uname -m'
i686

root@docker-emacs:~# docker run --rm -it arm32v7/ubuntu bash -c 'uname -m'
armv7l

Thanks a lot, I'll run some tests and report if I find weird things.

[Silex]

Oh btw, I'm not sure you need to fix this in qus... 64bits-only arm architectures are pretty rare no? I mean for me an entry in the wiki would be ok.