Unable to Authenticate against Registry on Buildkit (Kubernetes Drive)

[davejhahn]

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

I have build agents running in Kubernetes. When they run a build, it uses docker buildx create using the kubernetes drive, and then docker buildx build to run the build.

I ran into some issues because it was using anonymous authentication against Docker.io. The limit is 100 per 6 hours. My goal is to have buildkit authenticate against Docker.Io so there is no rate limit (using our licensed user and API auth token).

What I did:

• Created a Kubernetes secret (generic) with the docker.config file containing the auth token in the buildkit namespace
• Inject the volume mount on any pod container buildkitd in the buildkit namespace to /root/.docker
• Inject an ENV variable DOCKER_CONFIG=/root/.docker same as above
• Added --bootstrap to the docker buildx create so that I can add a sleep and access the buildkit pod as it is created.

I see:

• /root/.docker/config.json with the expected and validated auth
• ENV variable DOCKER_CONFIG set to /root/.docker

Prior to running the build, I obtain the rate limit and print it in the build logs (which uses Anonymous auth):

      TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
      curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest

What I observe is that after each build, the ratelmit decreases indicating it is using the Anonymous authentication.

I have also tried creating a .toml config file and passing it in with --config:

   [registry."docker.io"]
     config = "/root/.docker/config.json"

Using docker buiildx inspect to confirm that it is being passed and validating in /etc/ folder that the .toml file with the expected name is there and has the expected values.

Maybe I'm doing something wrong, most likely, but I am out of ideas. It seems that buildkit does not adhere to any sort of authentication provided.

Added information:

• Running Azure DevOps
• Each Build Job Creates a Keda Scaled Job for a new Build Agent Pod
• The build runs on the above pod, and then buildkit creates a subsequent pod in a different namespace for the build

I would also add that the creation of the buildkit pod (which also pulls from docker.io is NOT a problem). In this case, I've created a K8S secret in the buildkit namespace and added a pullsecret to the default service account in the buildkit namespace. I know this works because we were at a point where the rate limit was 0 and adding this immediately began working. But that only helps for the buildkit pod, not the builds within the buildkit pod.

Version information

docker.io/moby/buildkit:buildx-stable-1