Description
Contributing guidelines and issue reporting guide
- I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.
Well-formed report checklist
- I have found a bug that the documentation does not mention anything about my problem
- I have found a bug that there are no open or closed issues that are related to my problem
- I have provided version/information about my environment and done my best to provide a reproducer
Description of bug
Bug description
I have build agents running in Kubernetes. When they run a build, it uses docker buildx create using the kubernetes drive, and then docker buildx build to run the build.
I ran into some issues because it was using anonymous authentication against Docker.io. The limit is 100 per 6 hours. My goal is to have buildkit authenticate against Docker.Io so there is no rate limit (using our licensed user and API auth token).
What I did:
- Created a Kubernetes secret (generic) with the docker.config file containing the auth token in the buildkit namespace
- Inject the volume mount on any pod container buildkitd in the buildkit namespace to /root/.docker
- Inject an ENV variable DOCKER_CONFIG=/root/.docker same as above
- Added --bootstrap to the docker buildx create so that I can add a sleep and access the buildkit pod as it is created.
I see:
- /root/.docker/config.json with the expected and validated auth
- ENV variable DOCKER_CONFIG set to /root/.docker
Prior to running the build, I obtain the rate limit and print it in the build logs (which uses Anonymous auth):
TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest
What I observe is that after each build, the ratelmit decreases indicating it is using the Anonymous authentication.
I have also tried creating a .toml config file and passing it in with --config:
[registry."docker.io"]
config = "/root/.docker/config.json"
Using docker buiildx inspect to confirm that it is being passed and validating in /etc/ folder that the .toml file with the expected name is there and has the expected values.
Maybe I'm doing something wrong, most likely, but I am out of ideas. It seems that buildkit does not adhere to any sort of authentication provided.
Added information:
- Running Azure DevOps
- Each Build Job Creates a Keda Scaled Job for a new Build Agent Pod
- The build runs on the above pod, and then buildkit creates a subsequent pod in a different namespace for the build
I would also add that the creation of the buildkit pod (which also pulls from docker.io is NOT a problem). In this case, I've created a K8S secret in the buildkit namespace and added a pullsecret to the default service account in the buildkit namespace. I know this works because we were at a point where the rate limit was 0 and adding this immediately began working. But that only helps for the buildkit pod, not the builds within the buildkit pod.
Version information
docker.io/moby/buildkit:buildx-stable-1