How to properly mirror registries?

[EStork09]

I have a private registry that I am trying to get all pods to get proxied through. No matter what settings I seem to try for my mirrors, the configs show up in the file, but when I pull an image from a registry it doesn't matter, and the logs from containerd don't seem to acknowledge anything. Any help would be appreciated!

rke2 -v
rke2 version v1.29.4+rke2r1 (d30ec2a3fc17b931e2574fa8e7f082a48e8cfc0b)
go version go1.21.9 X:boringcrypto

mirrors:
  "docker.io":
    endpoint:
      - "https://harbor.example.com:443"
    rewrite:
      "(.*)": "docker/$1"
  "quay.io":
    endpoint:
      - "https://harbor.example.com:443"
    rewrite:
      "(.*)": "quay/$1"
  "ghcr.io":
    endpoint:
      - "https://harbor.example.com:443"
    rewrite:
      "(.*)": "ghcr/$1"

# File generated by rke2. DO NOT EDIT.

server = "https://registry-1.docker.io/v2"
capabilities = ["pull", "resolve", "push"]


[host]
[host."https://harbor.example.com:443/v2"]
  capabilities = ["pull", "resolve"]
  [host."https://harbor.example.com:443/v2".rewrite]
    "(.*)" = "docker/$1"

[brandond]

when I pull an image from a registry it doesn't matter

How are you pulling images? Are you trying to pull by hand using ctr or crictl, or just running a pod from one of the mirrored registries?

the logs from containerd don't seem to acknowledge anything.

The containerd.log should show it trying your endpoint and then falling back to the default endpoint if for some reason it does not work. Please attach the full containerd.log showing an image pull operation.

[EStork09]

How are you pulling images? Are you trying to pull by hand using ctr or crictl, or just running a pod from one of the mirrored registries?

Just normal deployment pointing at the mirrored registry.

The containerd.log should show it trying your endpoint and then falling back to the default endpoint if for some reason it does not work. Please attach the full containerd.log showing an image pull operation.

Explore-logs-2024-05-23 23 03 43.txt

I see it pull the image, but never try to reach out to anything other than docker.io (or just /repo/image:tag, you know, without the docker.io but is docker.io, not sure the right way to say that)

[brandond]

It looks to me like it's working, here's a log entry showing it trying both endpoints for an image from quay.io:

2024-05-23 22:59:03.755	time="2024-05-23T16:59:03.524494846-04:00" level=info msg="PullImage \"quay.io/opstree/redis-operator:v0.16.0\""
2024-05-23 22:59:06.591	time="2024-05-23T16:59:06.319577898-04:00" level=info msg="trying next host - response was http.StatusNotFound" host="zot.pineapplenest.com:443"
2024-05-23 22:59:06.841	time="2024-05-23T16:59:06.792409850-04:00" level=info msg="trying next host - response was http.StatusNotFound" host=quay.io

Note that the images won't LOOK like they're coming from your mirror, were you expecting to see them show up with different names after being pulled? Thats now how mirroring works, they're just retrieved from a different source, the pulled image names are not actually transformed.

[EStork09]

Yes, I was expecting the name to change, that's interesting, I am dumb, thank you for helping!

[EStork09]

So I guess it won't show when it has successfully pulled, only when there is an error? The image I was looking at was the oncall image, when that get pulls it doesn't say anything. Is there a way to validate the image was pulled from the mirror since it won't show anything? I assumed there would be some notification or something that it was pulled.

[brandond]

Containerd does not log what endpoint it pulled the image from at the default log level, no. It only logs the failed endpoint when the pull fails. You'd have to enable debug logging to see all the various http requests it makes when pulling an image.