musllinux/aarch64 builds timing out: suggestions for debugging?

[mdickinson]

Hi all,

I'm having trouble with musllinux / aarch64 wheel builds timing out on a GitHub Actions ubuntu-latest runner (with QEMU installed).
The tail end of the build before it's cancelled looks something like this:

Tue, 16 Aug 2022 13:59:45 GMT
Building cp37-musllinux_aarch64 wheel
[2449](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2562)
Tue, 16 Aug 2022 13:59:45 GMT
CPython 3.7 musllinux aarch64
[2450](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2563)
Tue, 16 Aug 2022 13:59:45 GMT

[2451](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2564)
Tue, 16 Aug 2022 13:59:45 GMT
Setting up build environment...
[2456](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2570)
Tue, 16 Aug 2022 13:59:46 GMT
                                                              ✓ 0.59s
[2457](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2571)
Tue, 16 Aug 2022 13:59:46 GMT
Building wheel...
[2458](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2572)
Tue, 16 Aug 2022 13:59:46 GMT
  
[2459](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2573)
Tue, 16 Aug 2022 13:59:46 GMT
      + rm -rf /tmp/cibuildwheel/built_wheel
[2460](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2574)
Tue, 16 Aug 2022 13:59:46 GMT
      + mkdir -p /tmp/cibuildwheel/built_wheel
[2461](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2575)
Tue, 16 Aug 2022 13:59:46 GMT
      + python -m pip wheel /project --wheel-dir=/tmp/cibuildwheel/built_wheel --no-deps
[2462](https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true#step:6:2576)
Tue, 16 Aug 2022 17:25:38 GMT
  Error: The operation was canceled.

My first thought was that I'd messed something up in the workflow or that there was a bug in the project, but now I'm seeing similar issues on a different project (albeit with a very similar build setup).

Any ideas what the issue might be, or suggestions about how to go about diagnosing and debugging? Has anyone else run into similar issues?

Here are the actual workflow runs, in case anyone wants to look:

• https://github.com/enthought/ibm2ieee/runs/7856723578?check_suite_focus=true
• https://github.com/enthought/scimath/runs/7862042960?check_suite_focus=true

And the workflows themselves:

• https://github.com/enthought/scimath/blob/main/.github/workflows/release-to-pypi.yml
• https://github.com/enthought/ibm2ieee/blob/main/.github/workflows/upload-to-pypi.yml

A key point is that both projects above have C extensions based on NumPy, so both need NumPy in the build environment. My best guess right now is that what's happening is that we end up building NumPy itself under QEMU emulation, and that that's so slow that the builds appear to be hanging.

[Czaki]

Currently, aarch64 builds are done using emulation, so they are much slower (I do not remember the factor).
I think that your assumption about numpy may be correct. Personally, I will try to build numpy wheels upload it to some public, controlled by you webserver and use PIP_EXTRA_INDEX_URL to pass it to build process.

Or wait on #1191

[mdickinson]

Thanks, @Czaki. This looks like the correct diagnosis. We re-ran the workflows in question after excluding musllinux builds and everything went much faster.