Any recommendation to avoid "Too many requests" limit with Python SDK

[skycaptain]

We're currently considering using Dagger for production. As our main language is Python, we want to write our Dagger modules in Python. At the moment, we're using the default Python SDK that Dagger provides. This SDK fetches python:3.11-slim from Docker Hub.

However, we've recently encountered the "Too Many Requests" limit on Docker Hub while trying to scale for production. We usually bypass this issue by proxying external images through our Artifactory server. The problem is, the image address appears to be hardcoded in the Python SDK. Therefore, we're seeking recommendations on how to properly scale this. Should we use a custom SDK and hard-code our proxied image? Given #6113, this doesn't seem to be an option for us at the moment, as we would be required to make parts about our infrastructure public.

For reference, see also #6486.

[skycaptain]

I realized I essentially answered my question in #6318: We could fork the python-sdk and configure the base image through dagger.json.

[helderco]

There’s an easier solution in v0.11, via pyproject.toml:

[tool.dagger]
base-image = "…"

[skycaptain]

That's awesome; I will definitely give it a try

[helderco]

The engine supports a mirrors setting. Have you tried that? Or doesn’t your Artifactory server work as a mirror?

[skycaptain]

Yes, I have and it's working great. However, I think we can only use this for CI, but not for local development as developers would forget to run a custom engine container with the patched engine.toml.