Error pulling the coredns image from a custom repository

[viluon]

What Happened?

This is a reopening of

• --image-repository + coredns pulling issue due to namespace removed #13324,

which was closed incorrectly.

Using a custom repository, e.g.

minikube \
    start \
    --memory=8192mb \
    --cpus=4 \
    --image-repository=remote-docker.artifactory.REDACTED

leads to an image pull error. Minikube requests coredns:v1.11.1 when it should be pulling coredns/coredns:v1.11.1 instead.

Attach the log file

 * minikube v1.34.0 on Ubuntu 22.04 (amd64)
* Automatically selected the docker driver
* Using image repository remote-docker.artifactory.dbgcloud.io
* Using Docker driver with root privileges
* Starting "minikube" primary control-plane node in "minikube" cluster
* Pulling base image v0.0.45 ...

<snip>

 stdout:
[init] Using Kubernetes version: v1.31.0
[preflight] Running pre-flight checks
[preflight] Pulling images required for setting up a Kubernetes cluster
[preflight] This might take a minute or two, depending on the speed of your internet connection
[preflight] You can also perform this action beforehand using 'kubeadm config images pull'
 	[WARNING Service-Kubelet]: kubelet service is not enabled, please run 'systemctl enable kubelet.service'
error execution phase preflight: [preflight] Some fatal errors occurred:
	[ERROR ImagePull]: failed to pull image remote-docker.artifactory.REDACTED/coredns:v1.11.1: failed to pull image remote-docker.artifactory.REDACTED/coredns:v1.11.1: Error response from daemon: manifest for remote-docker.artifactory.REDACTED/coredns:v1.11.1 not found: manifest unknown: The named manifest is not known to the registry.

Operating System

Ubuntu

Driver

Docker

[neolit123]

remote-docker.artifactory.REDACTED/coredns:v1.11.1

put it in remote-docker.artifactory.REDACTED/coredns:v1.11.1

it's something the coredns maintainers decided to do and kubeadm people decided to be backwards compat.

Image paths between the default registry.k8s.io and a custom repository specified using imageRepository may differ for backwards compatibility reasons. For example, one image might have a subpath at registry.k8s.io/subpath/image, but be defaulted to my.customrepository.io/image when using a custom repository.

https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/#custom-images

default registry: uses coredns subpath
custom registry: doesn't use coredns subpath

[viluon]

@neolit123 this discrepancy is a bug from my PoV. In a sandboxed environment where only access to an Artifactory virtual repo (mirror of some upstream like registry.k8s.io) is available, --image-repository doesn't work, even if the repository mirrors the upstream perfectly.

I'm happy to open an issue for kubeadm but I think this one should stay open until the behaviour matches that of the default registry.

[neolit123]

i'm not a fan but it cannot be changed anymore.
also it breaks users in both options, once e.g. etcd start using a subpath as well.

your fix is to:

• pass an override using the kubeadm config clusterconfiguration.dns.image with the image that you want.
• tag images in a custom way.

[hajnej]

Hey @neolit123, is there way how to pass that custom kubeadm config? The --extra-config does not support overriding the dns.image and there is not any other way how to override the kubeadm config created by minikube kubeadm bootstrapper.

minikube start --image-repository=remote-docker.PRIVATE_DOMAIN --extra-config='kubeadm.dns.image=remote-docker.PRIVATE_DOMAIN/coredns/coredns:1.11.1'
😄  minikube v1.34.0 on Ubuntu 22.04
✨  Automatically selected the docker driver. Other choices: qemu2, ssh

❌  Exiting due to MK_USAGE: Sorry, the kubeadm.dns.image parameter is currently not supported by --extra-config

[neolit123]

that's a Q for the minikube maintainers.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[OhMyMndy]

I made a dirty fix so I can continue using my Nexus repository proxy, I have Nginx in front of the registry rewriting the URL for coredns...

server {
  listen 80;
  server_name oci.nexus.home.ohmymndy.com;

  location / {
    proxy_pass http://nexus:8081/repository/oci/;
  }
  
  location ~ ^/v2/coredns/(.*)$ {
    proxy_pass http://nexus:8081/repository/oci/v2/coredns/coredns/$1$is_args$args;
  }
}

https://github.com/OhMyMndy/homelab/blob/8e506ad1d51d3c472f11c77048359ceb3fe0e021/nexus/nexus-dockerhub.conf#L10

[viluon]

/remove-lifecycle stale

[bbodenmiller]

Anyone find a workaround here that is not changing path of image on custom registry?

Upstream PR that implemented coredns/coredns with default registry only: kubernetes/kubernetes#102502

Even retagging image locally doesn't seem to work:

docker pull remote-docker.artifactory.REDACTED/coredns/coredns:v1.12.0 && docker tag remote-docker.artifactory.REDACTED/coredns/coredns:v1.12.0 remote-docker.artifactory.REDACTED/coredns:v1.12.0
minikube start --image-repository="remote-docker.artifactory.REDACTED"
...
	[ERROR ImagePull]: failed to pull imageremote-docker.artifactory.REDACTED/coredns:v1.12.0: failed to pull image remote-docker.artifactory.REDACTED/coredns:v1.12.0: Error response from daemon: manifest for remote-docker.artifactory.REDACTED/coredns:v1.12.0 not found: manifest unknown: manifest unknown

[bbodenmiller]

A workaround: after hitting [ERROR ImagePull]: failed to pull image remote-docker.artifactory.REDACTED/coredns:v1.12.0 is to pull and put image in minikube where it wants it then run start command again:

docker exec -it minikube /bin/bash -c "docker pull remote-docker.artifactory.REDACTED/coredns/coredns:v1.12.0 && docker tag remote-docker.artifactory.REDACTED/coredns/coredns:v1.12.0 remote-docker.artifactory.REDACTED/coredns:v1.12.0"
minikube start --image-repository="remote-docker.artifactory.REDACTED" --binary-mirror="https://remote-docker-bins.artifactory.REDACTED/artifactory/dl.k8s.io" --preload=false

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale