How do I get buildctl to load the Kubernetes CA from /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

[crouth-redge]

I am trying this solution for buildkit on unprivileged gitlab-runners (on GKE autopilot) https://forum.gitlab.com/t/migrating-from-kaniko-to-buildkit-still-need-elevated-permissions/128648/9

Everything is setup - but the last hurdle seems to be getting buildctl to use /var/run/secrets/kubernetes.io/serviceaccount/ca.crt to trust the certificate that the remote buildkitd uses.

I've tried passing --tlscacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt as well as copying it to /etc/ssl/certs

Unfortunately AI addled search is helping nothing - and every result is confused with pushing to insecure registries or setting up the buildkitd daemon

I need to trust the CA from the buildctl side - how do I do that?

  script:
    - |
      buildctl \
        --addr tcp://${BUILDCTL_HOST}:${BUILDCTL_PORT} \
        --tlscacert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      build \
        --frontend dockerfile.v0 \
        --local context=. \
        --local dockerfile=. \
        --opt build-arg:REGISTRY_URL="${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}" \
        --export-cache type=registry,ref=$CACHE_IMAGE \
        --import-cache type=registry,ref=$CACHE_IMAGE \
        --output type=image,name=$CI_REGISTRY_IMAGE/test:$CI_COMMIT_SHA,push=true

[crouth-redge]

Worked out that there's a hard requirement on mutual TLS here when using certs - so I need "client" certs.

Mounting the same secret to the jobs did the trick.

This needs to be added under [runners.kubernetes]

          [[runners.kubernetes.volumes.secret]]
            name = "gitlab-runner-buildkit-daemon-certs"
            mount_path = "/certs"

And then this is how the job works:

build-cached-rootless:
  image:
    name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/moby/buildkit:v0.23.2
    entrypoint: [""]
  stage: build
  variables:
    BUILDCTL_HOST: gitlab-runner
    BUILDCTL_NAMESPACE: gitlab
    CLUSTER_DOMAIN: cluster.local
    BUILDCTL_PORT: "1234"
    CACHE_IMAGE: $CI_REGISTRY_IMAGE:cache
  before_script:
    - mkdir -p ~/.docker
    - |
      echo "{
        \"auths\": {
          \"${CI_REGISTRY}\": {
            \"auth\": \"$(printf "%s:%s" "${CI_REGISTRY_USER}" "${CI_REGISTRY_PASSWORD}" | base64 | tr -d '\n')\"
          },
          \"$(echo -n $CI_DEPENDENCY_PROXY_SERVER | awk -F[:] '{print $1}')\": {
            \"auth\": \"$(printf "%s:%s" ${CI_DEPENDENCY_PROXY_USER} "${CI_DEPENDENCY_PROXY_PASSWORD}" | base64 | tr -d '\n')\"
          }
        }
      }" > ~/.docker/config.json
  script:
    - |
      buildctl \
        --addr tcp://${BUILDCTL_HOST}.${BUILDCTL_NAMESPACE}.svc.${CLUSTER_DOMAIN}:${BUILDCTL_PORT} \
        --tlscacert /certs/ca.crt \
        --tlscert /certs/tls.crt \
        --tlskey /certs/tls.key \
        --tlsservername ${BUILDCTL_HOST}.${BUILDCTL_NAMESPACE}.svc.${CLUSTER_DOMAIN} \
      build \
        --frontend dockerfile.v0 \
        --local context=. \
        --local dockerfile=. \
        --opt build-arg:REGISTRY_URL="${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}" \
        --export-cache type=registry,ref=$CACHE_IMAGE \
        --import-cache type=registry,ref=$CACHE_IMAGE \
        --output type=image,name=$CI_REGISTRY_IMAGE/test:$CI_COMMIT_SHA,push=true