Add `UV_PYTHON_DOWNLOADS_JSON_URL` to set custom managed python sources #10939

MeitarR · 2025-01-24T15:40:43Z

Summary

Add an option to overwrite the list of available Python downloads from a local JSON file by using the environment variable UV_PYTHON_DOWNLOADS_JSON_URL

as an experimental support for providing custom sources for Python distribution binaries #8015

related #10203

I probably should make the JSON to be fetched from a remote URL instead of a local file.
please let me know what you think and I will modify the code accordingly.

Test Plan

normal run

root@75c66494ba8b:/# /code/target/release/uv python list
cpython-3.14.0a4+freethreaded-linux-x86_64-gnu    <download available>
cpython-3.14.0a4-linux-x86_64-gnu                 <download available>
cpython-3.13.1+freethreaded-linux-x86_64-gnu      <download available>
cpython-3.13.1-linux-x86_64-gnu                   <download available>
cpython-3.12.8-linux-x86_64-gnu                   <download available>
cpython-3.11.11-linux-x86_64-gnu                  <download available>
cpython-3.10.16-linux-x86_64-gnu                  <download available>
cpython-3.9.21-linux-x86_64-gnu                   <download available>
cpython-3.8.20-linux-x86_64-gnu                   <download available>
cpython-3.7.9-linux-x86_64-gnu                    <download available>
pypy-3.10.14-linux-x86_64-gnu                     <download available>
pypy-3.9.19-linux-x86_64-gnu                      <download available>
pypy-3.8.16-linux-x86_64-gnu                      <download available>
pypy-3.7.13-linux-x86_64-gnu                      <download available>

empty JSON file

root@75c66494ba8b:/# export UV_PYTHON_DOWNLOADS_JSON_URL=/code/crates/uv-python/my-download-metadata.json 
root@75c66494ba8b:/# cat $UV_PYTHON_DOWNLOADS_JSON_URL 
{}
root@75c66494ba8b:/# /code/target/release/uv python list
root@75c66494ba8b:/#

JSON file with valid version

root@75c66494ba8b:/# export UV_PYTHON_DOWNLOADS_JSON_URL=/code/crates/uv-python/my-download-metadata.json 
root@75c66494ba8b:/# cat $UV_PYTHON_DOWNLOADS_JSON_URL 
{
  "cpython-3.11.9-linux-x86_64-gnu": {
    "name": "cpython",
    "arch": {
      "family": "x86_64",
      "variant": null
    },
    "os": "linux",
    "libc": "gnu",
    "major": 3,
    "minor": 11,
    "patch": 9,
    "prerelease": "",
    "url": "https://github.com/astral-sh/python-build-standalone/releases/download/20240814/cpython-3.11.9%2B20240814-x86_64-unknown-linux-gnu-install_only_stripped.tar.gz",
    "sha256": "daa487c7e73005c4426ac393273117cf0e2dc4ab9b2eeda366e04cd00eea00c9",
    "variant": null
  }
}
root@75c66494ba8b:/# /code/target/release/uv python list
cpython-3.11.9-linux-x86_64-gnu    <download available>
root@75c66494ba8b:/#

Remote Path

root@75c66494ba8b:/# export UV_PYTHON_DOWNLOADS_JSON_URL=http://a.com/file.json 
root@75c66494ba8b:/# /code/target/release/uv python list
error: Remote python downloads JSON is not yet supported, please use a local path (without `file://` prefix)

zanieb · 2025-01-24T22:32:17Z

crates/uv-python/src/downloads.rs

-            // linked), so we pretend they don't exist. https://github.com/astral-sh/uv/issues/4242
-            .filter(|download| download.key.libc != Libc::Some(target_lexicon::Environment::Musl))
+        PYTHON_DOWNLOADS.get_or_init(|| {
+            if let Ok(json_path) = std::env::var("UV_PYTHON_DOWNLOADS_JSON_PATH") {


Note this variable will need to be defined in crates/uv-static/src/env_vars.rs

I wonder if we should call it _URL or _URI? We can assume a local path and error with an informative message about the lack of remote file support for now, but we should have a forward-looking name.

We also need to be considerate about how the fetch is implemented, e.g., if we're making a network request it should probably be async which would mean this this function needs to be async (which I would be disappointed by). What kind of change would we need to make to avoid that?

(I don't think we need remote file support to add this feature — but I would like to know how invasive it would be)

A remote file also introduces the complexity of caching 🤔 though perhaps we could avoid that since installs don't happen that often.

looks like it won't be that bad to make it async.
we would also need to make the following async:

iter_downloads

from_request

PytestRequest::new

But maybe we could use tokio::runtime::Handle::current().block_on or construct a new runtime for the request only.

zanieb · 2025-01-24T22:34:52Z

crates/uv-python/src/downloads.rs

+                let json_downloads: HashMap<String, JsonPythonDownload> = serde_json::from_reader(file)
+                    .unwrap_or_else(|e| {
+                        panic!("Failed to parse JSON file at {json_path}: {e}");
+                    });


Maybe a bit of a silly question, but is there a reason we shouldn't deserialize directly into ManagedPythonDownload? Why the intermediate type?

TBH, that's what the AI suggested, but I had the same thought as you, so I tried to derive the Deserialize on ManagedPythonDownload and then on PythonInstallationKey. I understood that target_lexicon's enums do not implement the Deserialize trait, and even if I overcome this, the conversion from JSON will not be as is. So the only way to improve it (the way I see it) is to implement Deserialize on our own and define the JSON structs inside the implementation.

If you see a better way, please let me know

It looks like there's a serde feature for target_lexicon, did you try that?

What part is not "as-is"? Like some custom mapping of architectures or something?

It looks like there's a serde feature for target_lexicon, did you try that?

from the code, I understood that they only implemented it for the Triple struct, so it not useful here :(

What part is not "as-is"? Like some custom mapping of architectures or something?

yes, it's small details, like the implementation that will need special implementation, or the arch. the main problem here is the structs of target_lexicon

zanieb · 2025-01-24T22:36:26Z

crates/uv-python/src/downloads.rs

+
+                let json_downloads: HashMap<String, JsonPythonDownload> = serde_json::from_reader(file)
+                    .unwrap_or_else(|e| {
+                        panic!("Failed to parse JSON file at {json_path}: {e}");


A panic is probably not acceptable here. We'd need to handle this gracefully, i.e., by surfacing the error to the user.

MeitarR · 2025-01-30T18:44:52Z

@zanieb is there something left for me to do to get this merged?

…URL` and moved it to `crates/uv-static/src/env_vars.rs`

zanieb · 2025-02-03T21:55:32Z

Just need to find the time to review it carefully and make sure we're happy maintaining / supporting this in perpetuity.

zanieb self-assigned this Jan 24, 2025

zanieb reviewed Jan 24, 2025

View reviewed changes

MeitarR requested a review from zanieb January 25, 2025 21:46

MeitarR force-pushed the uv-python-downloads-json-file branch from 76c6528 to 1af245c Compare January 25, 2025 21:48

MeitarR changed the title ~~custom sources for Python distribution binaries~~ Add UV_PYTHON_DOWNLOADS_JSON_URL to set custom managed python sources Jan 27, 2025

MeitarR force-pushed the uv-python-downloads-json-file branch from 1af245c to dedc024 Compare January 30, 2025 19:24

MeitarR added 3 commits January 30, 2025 21:27

Refactor Python downloads handling to support JSON configuration

9f0ab4b

changed UV_PYTHON_DOWNLOADS_JSON_PATH to `UV_PYTHON_DOWNLOADS_JSON_…

d53923b

…URL` and moved it to `crates/uv-static/src/env_vars.rs`

surface the errors to the user

93e045c

MeitarR force-pushed the uv-python-downloads-json-file branch from dedc024 to 93e045c Compare January 30, 2025 19:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add `UV_PYTHON_DOWNLOADS_JSON_URL` to set custom managed python sources #10939

Add `UV_PYTHON_DOWNLOADS_JSON_URL` to set custom managed python sources #10939

MeitarR commented Jan 24, 2025 •

edited

Loading

zanieb Jan 24, 2025

zanieb Jan 24, 2025

zanieb Jan 24, 2025

MeitarR Jan 25, 2025

zanieb Jan 24, 2025

MeitarR Jan 25, 2025 •

edited

Loading

zanieb Jan 27, 2025

zanieb Jan 27, 2025

MeitarR Jan 27, 2025

zanieb Jan 24, 2025 •

edited

Loading

MeitarR commented Jan 30, 2025

zanieb commented Feb 3, 2025

Add UV_PYTHON_DOWNLOADS_JSON_URL to set custom managed python sources #10939

Are you sure you want to change the base?

Add UV_PYTHON_DOWNLOADS_JSON_URL to set custom managed python sources #10939

Conversation

MeitarR commented Jan 24, 2025 • edited Loading

Summary

Test Plan

normal run

empty JSON file

JSON file with valid version

Remote Path

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MeitarR Jan 25, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zanieb Jan 24, 2025 • edited Loading

Choose a reason for hiding this comment

MeitarR commented Jan 30, 2025

zanieb commented Feb 3, 2025

Add `UV_PYTHON_DOWNLOADS_JSON_URL` to set custom managed python sources #10939

Add `UV_PYTHON_DOWNLOADS_JSON_URL` to set custom managed python sources #10939

MeitarR commented Jan 24, 2025 •

edited

Loading

MeitarR Jan 25, 2025 •

edited

Loading

zanieb Jan 24, 2025 •

edited

Loading