Skip to content

Unable to Authenticate against Registry on Buildkit (Kubernetes Drive) #5762

Open
@davejhahn

Description

@davejhahn

Contributing guidelines and issue reporting guide

Well-formed report checklist

  • I have found a bug that the documentation does not mention anything about my problem
  • I have found a bug that there are no open or closed issues that are related to my problem
  • I have provided version/information about my environment and done my best to provide a reproducer

Description of bug

Bug description

I have build agents running in Kubernetes. When they run a build, it uses docker buildx create using the kubernetes drive, and then docker buildx build to run the build.

I ran into some issues because it was using anonymous authentication against Docker.io. The limit is 100 per 6 hours. My goal is to have buildkit authenticate against Docker.Io so there is no rate limit (using our licensed user and API auth token).

What I did:

  • Created a Kubernetes secret (generic) with the docker.config file containing the auth token in the buildkit namespace
  • Inject the volume mount on any pod container buildkitd in the buildkit namespace to /root/.docker
  • Inject an ENV variable DOCKER_CONFIG=/root/.docker same as above
  • Added --bootstrap to the docker buildx create so that I can add a sleep and access the buildkit pod as it is created.

I see:

  • /root/.docker/config.json with the expected and validated auth
  • ENV variable DOCKER_CONFIG set to /root/.docker

Prior to running the build, I obtain the rate limit and print it in the build logs (which uses Anonymous auth):

      TOKEN=$(curl "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
      curl --head -H "Authorization: Bearer $TOKEN" https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest

What I observe is that after each build, the ratelmit decreases indicating it is using the Anonymous authentication.

I have also tried creating a .toml config file and passing it in with --config:

   [registry."docker.io"]
     config = "/root/.docker/config.json"

Using docker buiildx inspect to confirm that it is being passed and validating in /etc/ folder that the .toml file with the expected name is there and has the expected values.

Maybe I'm doing something wrong, most likely, but I am out of ideas. It seems that buildkit does not adhere to any sort of authentication provided.

Added information:

  • Running Azure DevOps
  • Each Build Job Creates a Keda Scaled Job for a new Build Agent Pod
  • The build runs on the above pod, and then buildkit creates a subsequent pod in a different namespace for the build

I would also add that the creation of the buildkit pod (which also pulls from docker.io is NOT a problem). In this case, I've created a K8S secret in the buildkit namespace and added a pullsecret to the default service account in the buildkit namespace. I know this works because we were at a point where the rate limit was 0 and adding this immediately began working. But that only helps for the buildkit pod, not the builds within the buildkit pod.

Version information

docker.io/moby/buildkit:buildx-stable-1

Metadata

Metadata

Assignees

No one assigned

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions