Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Python 3.11 to the build matrix #938

Closed
wants to merge 1 commit into from

Conversation

jmarshall
Copy link
Member

@jmarshall jmarshall commented Nov 23, 2023

I have no idea what all is involved in updating this and perhaps doing initial package builds on the bulk branch, but this PR provides a starting point for this long-overdue update. Python 3.12 was released almost two months ago, so it is embarrassing that bioconda is still not building packages for the previous year's release, Python 3.11.

This is tracked as bioconda/bioconda-recipes#37805. That links to conversation on #878 which indicates that bioconda has been waiting for conda-forge to include 3.11. It appears that conda-forge's migration was completed in September.

@jmarshall jmarshall changed the title Add Python 3.11 to the build matrix feat: Add Python 3.11 to the build matrix Nov 23, 2023
@hasindu2008
Copy link

I hope bioconda soon supports Python 3.11. I have got a request to get pyslow5 for Python 3.11 on conda hasindu2008/slow5lib#83.

@jmarshall
Copy link
Member Author

@bioconda/core: Please comment as to whether bioconda intends to support Python 3.11.

@bgruening
Copy link
Member

Hi John. Yes, we intend to support Python 3.11 and 3.12. However, we are currently busy rolling out the ARM builds and in parallel the BioConductor release. This keeps us and the CI already busy. Realistically we can only care about 3.11/12 in January. Sorry.

@jmarshall
Copy link
Member Author

@bgruening: Thanks for the information. I suggest you post an update on bioconda/bioconda-recipes#37805, bioconda/bioconda-recipes#33333, and/or bioconda/bioconda-recipes#37068. I ask not for myself, but for the numerous bioconda users regularly asking myself and other maintainers of python-version-specific packages why our packages are not available on Python 3.11.

In general, the lack of information from the core cabal is disappointing. When bioconda/bioconda-recipes#33333 and bioconda/bioconda-recipes#37068 were pinned, I was hopeful that they would become useful conduits, but they have largely not been used.

@bgruening
Copy link
Member

I agree, we need to improve our communications.
Maybe someone wants to join us and help communicate what we are currently doing.

@corneliusroemer
Copy link
Member

I'd be happy to help a little with communication

@corneliusroemer
Copy link
Member

@bgruening gentle ping to ask about status of bioconductor release, ARM feature and Python 3.11/3.12 support. @jmarshall mentioned that bioconductor release seems to be complete.

Your past updates were very helpful, I'm hoping you can give another short summary of where things stand so the community at large is somewhat in the loop.

@daler
Copy link
Member

daler commented Jan 16, 2024

@bgruening can chime in if there's anything I'm missing here, but here's a snapshot of the current state and short-term goals:

Current goal is to get everything cleaned up and in shape to support large-scale updating for ARM and Python 3.11 specifically regarding the build-failure yamls and their interaction with the bulk and master branches. For example, right now removing a build-failure yaml is insufficient to trigger a rebuild; we need to work out the right process for having many people work on build failures simultaneously. This requires the technical component (actually writing the infrastructure code) but also the sociological component (how, specifically, do we best coordinate across contributors without stepping on each others' toes)? This is important for both ARM and Python 3.11 and of course any future migrations/updates. The “practice run” for this, once the infra is in place, will be finalizing the remaining Bioconductor packages.

There are a last few remaining details to be taken care of with ARM containers to avoid arm64 clobbering amd6. This involves quay.io, docker manifests, and making sure all the infrastructure is updated to handle all of that. Then we’ll kick off some opt-in ARM builds to make sure everything works in practice.

Then it’s on to Python 3.11. Hopefully the infrastructure will be in place there for rapid building on bulk, kicking out build failures to the wiki, and coordinating with the community to fix everything. My dream is for the 3.11 to go quickly and smoothly. Ideally, 3.12 could follow shortly after. And by then hopefully we'll have worked out the kinks on ARM, and can scale up those builds as well.

The challenging part in all of this is that to make meaningful progress on infrastructure components like ARM containers and the build-failure process, we need to load everything into mental RAM. That is difficult to do when we all have day jobs, many of us are trying run our respective labs and working on this in our (increasingly rare) spare time. So then documentation becomes a vital part of all of this to minimize the mental-RAM-loading time (e.g. https://github.com/bioconda/bioconda-docs/pull/16/files). The docs are certainly helping. Ideally, we would document everything extensively enough so the community can jump in and help out and alleviate the bandwidth issue. We’re not there yet,but we're getting there.

We of course also need to be better about the decision-making process, planning process, and documenting all of the various processes and moving parts to make this all more transparent to the community. Developing that documentation and communication takes away from making progress on working on infrastructure, so it's a balance.

None of this helps directly with Python 3.11, I know. Just trying to give some insight into how we're trying to improve all the internal processes and make it better for everyone over the long term. Rest assured that 3.11 is definitely in the works.

@jmarshall
Copy link
Member Author

@daler: Thanks for your comments. Posting them here does not really reach the audience who needs to hear them; wider communication channels might be gitter or the pinned announcement and roadmap issues.

Past Python migrations have been done without the benefit of the build-failure infrastructure. So — volunteer time issues aside — there does not seem to be a technical reason to further delay this overdue migration.

@daler
Copy link
Member

daler commented Jan 16, 2024

@jmarshall agreed on both counts.

The current mechanism, where Py 3.11 builds run on bulk branch in topological sort across workers, requires heavy time investment by a single person over a couple of weeks. This is how all previous migrations have been done. Nobody currently (or regularly) has bandwidth to manage that nowadays, hence the refactor for better scaling. So not a technical limitation, but rather a "personnel" limitation.

Also build python-version-specific packages for Python 3.11.*, now that
conda-forge's migration is complete. Update to current conda-forge-pinning
and update our numpy pinning to align with conda-forge's.
@corneliusroemer
Copy link
Member

Now that aarch64 is in the works (big congrats 🎉 ), what's the timeline for progress on Python 3.11

@mbargull
Copy link
Member

what's the timeline for progress on Python 3.11

I'm meant to start this a few weeks back actually, but got sick (multiple times 🤦). This is high on my prio list and should start next week.
(I'll also re-evaluate the status of 3.12's migration on conda-forge and may do 3.12 at the same or shortly after in case nothing major is missing for us.)

@daler
Copy link
Member

daler commented Mar 21, 2024

thanks @mbargull, hope you're feeling better soon. @corneliusroemer as mentioned previously this sort of updating is currently personnel-intensive, and Marcel will be the one handling it this round. We have some pieces in place to better democratize the process and speed up future migrations which we will be testing here.

@corneliusroemer
Copy link
Member

Excellent news, thank you both for the update!

aliciaaevans pushed a commit that referenced this pull request Jun 3, 2024
Amongst others, this includes:
- Update default minimum system targets to
- `glibc 2.17`/CentOS 7 ( \*needs recipes to include new `- {{
stdlib("c") }}` dep in `requirements/build`)
    - macOS 10.13
- Python 3.11 and 3.12

closes gh-938, gh-988

---------

Signed-off-by: Marcel Bargull <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants