Can we specify the order of the build regarding Python versions?

[TonyXiang8787]

We are thinking about to refactor our project to use ctypes based interface. In that way we only need to build once per platform. We are studying the official example.

One thing we noticed is that the build always started at the lowest Python version (3.6 in case of default). If we specify something in the config file like below. The build will start from cp36-manylinux_x86_64.

[tool.cibuildwheel.linux]
build = ["cp*-manylinux_x86_64", "cp*-manylinux_aarch64"]
archs = ["x86_64", "aarch64"]

This can be a problem, because some of our build dependencies do not support older Python versions. If we could specify the build order to build latest Python version first (3.11), this would not be a problem. Because when it is the turn for older Python version, there is already a compatible wheel file (e.g. py3-none-manylinux_2_17_x86_64) and the build step will be skipped. We could then include all the Python versions in the CI to test the built wheel file with all Python versions.

How can we specify that the order of build should be to build the latest Python version first?

[Czaki]

As you need only one run per platform, you could use build = ["cp36-manylinux_x86_64", "cp38-manylinux_aarch64"] etc. As it precisely specifies the python version to be used.

[TonyXiang8787]

@Czaki thanks for your reply. We were thinking about your suggestion. It will indeed the build the specified version. But it will not test the wheel with all Python versions. So that would not work completely.

I have briefly read the code base. I think it would not be difficult to sort the build configuration based on Python version one way or the other. We then need to add a new option like reverse-build-order or something. May I try to contribute a PR here?

[Czaki]

I have mixed feelings about this request, as you want to create a package that could be installed from the wheel but not from sdist. And this is not correct for me. But @joerick @YannickJadoul and @henryiii does not provide their opinion.

Maybe wait one or two days (as they are directly called now) and if they do not provide feedback please open PR.

[henryiii]

It's usually not needed. And in this case, why can't you just run cibuildwheel twice; once to build with the newest version, then once to test the built wheel on all the older versions?

[TonyXiang8787]

@henryiii thank you for your suggestion. So in the 2nd time when cibuildwheel is invoked will it look at the target folder (e.g. wheelhouse) to see if there is already a compatible built wheel? I can image this will work for Windows/Mac. But Linux builds happen in a container. How can the script inside container know that there is already a compatible wheel in the host OS?

I will have a try to see what will happen.

[joerick]

I concur with @Czaki's mixed feelings about this - building wheels where the sdist cannot install successfully isn't really how Python packaging is supposed to work. That said, I'm not trying to put up roadblocks to motivated people that want to make stuff work. So I'd be happy to support workarounds but would rather not require the creation of a new option/API (which incurs ongoing maintenance/testing burden).

Sorry I can't think of anything more specific. On a semi-side note, I do wonder in moments like these that a tool citestwheel would be useful, and might have other uses. But perhaps more work than is worth it for this use-case alone.

[TonyXiang8787]

I concur with @Czaki's mixed feelings about this - building wheels where the sdist cannot install successfully isn't really how Python packaging is supposed to work. That said, I'm not trying to put up roadblocks to motivated people that want to make stuff work. So I'd be happy to support workarounds but would rather not require the creation of a new option/API (which incurs ongoing maintenance/testing burden).

Sorry I can't think of anything more specific. On a semi-side note, I do wonder in moments like these that a tool citestwheel would be useful, and might have other uses. But perhaps more work than is worth it for this use-case alone.

Hi @joerick, thanks for your reply. I can understand that we are supposed to make both wheel and sdist work. In practice, however, due to the complexity of the build process we do not expect the users to install our packages from source, nor do we provide an sdist in the PyPI. Those who would like to build the package from source have to follow the complete build guide in our documentation anyway to prepare the build environment (toolchains, dependencies) and checkout the source code from GitHub.

Thank you for your openness to support workarounds. I will first try the suggestion of @henryiii to see if that will work.