Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fork actions/setup-python as Quansight-Labs/setup-python #107

Open
colesbury opened this issue Nov 4, 2024 · 10 comments
Open

Fork actions/setup-python as Quansight-Labs/setup-python #107

colesbury opened this issue Nov 4, 2024 · 10 comments
Assignees
Labels
enhancement New feature or request

Comments

@colesbury
Copy link
Contributor

colesbury commented Nov 4, 2024

While we wait for GitHub to review the PRs to support free-threaded Python (actions/setup-python#973, actions/python-versions#319), Quansight Labs can host a fork of setup-python that supports it. Here are the steps:

python-versions

  1. Fork actions/python-versions into the Quansight-Labs org. Make sure GitHub actions are enabled in the fork.

  2. Merge the five commits from the colesbury/python-versions quansight branch into the main branch of the new fork. In addition to the changes from my upstream PR, this branch has a few modifications to make it work outside of the actions organization, and to merge versions from upstream actions/python-versions (so that we don't have to rebuild every single Python version).

  3. Run the "Build Python Package" step at https://github.com/Quansight-Labs/python-versions/actions/workflows/build-python-packages.yml with the following options. This will take ~30 minutes:

    Branch: main
    Python version: 3.13.0
    Whether to publish releases: Yes (IMPORTANT!)
    CPython threading build modes: freethreaded
    Platforms: ubuntu-20.04,ubuntu-22.04,ubuntu-24.04,macos-13_x64,macos-14_arm64,windows-2019_x64,windows-2019_x86

(Only the "publish releases" needs to be changed -- the other options should be the same as the defaults)

This will take about 30 minutes and automatically publish a 3.13.0 release on the fork.

  1. Merge the automatically opened PR once the build process completes. This will update the versions-manifest.json file.

setup-python

  1. Fork actions/setup-python into the Quansight-Labs org. Make sure GitHub actions are enabled in the fork.
  2. Merge the two commits from the colesbury/setup-python quansight branch into the main branch of the new fork. In addition to the changes from my upstream PR, this branch changes the organization (for versions-manifest.json) to Quansight-Labs.
  3. Create a release (from https://github.com/Quansight-Labs/setup-python/releases). Name it something like v5.3.1, so that the major version (v5) stays the same as in upstream. This will automatically set the v5 tag as long.

Usage

Projects should be able to swap actions/setup-python with Quansight-Labs/setup-python and hopefully everything will "just work". Free threaded Pythons will be pulled from Quansight-Labs/python-versions while non-free threaded versions should continue working without any extra changes.

cc @rgommers

@rgommers
Copy link
Member

rgommers commented Nov 5, 2024

Wow, you've done almost everything already, thanks @colesbury!

I have created the two forks (https://github.com/Quansight-Labs/python-versions, https://github.com/Quansight-Labs/setup-python) under this org with the right config, and added @Quansight-Labs/free-threaded-team and @colesbury with maintain permissions.

@andfoy volunteered to take point on the maintenance. @andfoy can you take the next steps here in both repos?

@rgommers
Copy link
Member

rgommers commented Nov 5, 2024

@andfoy you may have to review the enabled and disabled actions in more detail. I disabled the ones that don't seem relevant (some were already disabled after forking), while the ones that may be needed for a release I left enabled. From https://github.com/Quansight-Labs/setup-python/actions/:

Image

@rgommers rgommers added the enhancement New feature or request label Nov 5, 2024
@colesbury
Copy link
Contributor Author

colesbury commented Nov 5, 2024

I'll create pull requests in the forks to make the process a bit easier.

@andfoy
Copy link
Member

andfoy commented Nov 5, 2024

I've just published Quansight-Labs/setup-python v5.3.1, I'll test it over at SciPy. Thanks a lot for your help @colesbury

@andfoy
Copy link
Member

andfoy commented Nov 5, 2024

I confirm it is working as expected: https://github.com/andfoy/scipy/actions/runs/11692095380/job/32560698824

@jimkring
Copy link
Contributor

jimkring commented Nov 5, 2024

It's working well for me, too. I tested on ubuntu-latest and windows-latest and no functional problems with the installations.

I did however notice an issue on Window with lots of log messages (dir listings of installed files).

@ngoldbaum
Copy link
Collaborator

Converting NumPy's CI was straightforward: numpy/numpy#27707

@greglucas
Copy link

Thanks for all the work in putting this together! This was extremely easy to drop in and get running.

I'm running into an issue on the windows-latest runner, where it looks like I might not be getting all of the underlying header files linked in properly. Many similar errors to this:

  C:/ProgramData/mingw64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/13.2.0/../../../../x86_64-w64-mingw32/bin/ld.exe: lib_fortranobject.a.p/46e04f6dbed83ac7b17b5e6c64deae3b55dde6f0_.._.._f2py_src_fortranobject.c.obj:fortranobject.:(.text+0x2c3d): undefined reference to `__imp__Py_MergeZeroLocalRefcount'

the linking works fine on linux and macos, and the windows-latest + 3.13 build, it is just the windows-latest + 3.13t that fails in the installation. This is a project using meson and f2py for wrapping some legacy Fortran code. Any thoughts from anyone on what could be going on here, whether I need to add another flag somewhere or if this is something in the action or numpy windows wheel that needs to be updated?

PR: SWxTREC/pymsis#61
link to failed job: https://github.com/greglucas/pymsis/actions/runs/11866957177/job/33074401949

@rgommers
Copy link
Member

@greglucas that looks like mesonbuild/meson#13851, which is MinGW-specific and isn't yet included in a Meson release (hopefully 1.6.1 will appear soon). In SciPy we're currently working around that like so (linking all the pieces that pull in Meson from GitHub for Windows cp313t wheel builds):

@greglucas
Copy link

Fantastic, thanks @rgommers! That looks like it fixed the problem and the windows free threading wheels are building again. Thanks again everyone for the work in making this so seamless and supporting so many projects in this transition.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

6 participants