[DOC] - Missing documentation on how to external repo via registries.yaml

[alex-ioma]

Environmental Info:
RKE2 Version: 1.28.10+rke2r1

Node(s) CPU architecture, OS, and Version: N/A

Cluster Configuration: 3 master + 3 workers

Describe the bug:

Apologies for opening this ticket but I could not find a way to add GCP Artifact Registry to the RKE2 containerd runtime via the registries.yaml file.

Steps To Reproduce:

• Create a Service Account Key on GCP/IAM with access to GCP Artifact Registry
• Test registry access with said key (it correctly pull from local)
• Try and add key but cannot figure out which how to complete the registries.yaml config file.

Expected behavior:

Be able to pull images via crictl or inside k8s.

Actual behavior:

Pull doesn't work - failing with:
Failed to pull image "europe-docker.pkg.dev/cm-controls/lgm/lgm-hmi:v0.1.0-alpha": failed to pull and unpack image "europe-docker.pkg.dev/cm-controls/lgm/lgm-hmi:v0.1.0-alpha": failed to resolve reference "europe-docker.pkg.dev/cm-controls/lgm/lgm-hmi:v0.1.0-alpha": failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://europe-docker.pkg.dev/v2/token?scope=repository%3Acm-controls%2Flgm%2Flgm-hmi%3Apull&service=europe-docker.pkg.dev: 403 Forbidden
Additional context / logs:

It's likely an error on my end on how to properly configure registries.yaml but I feel the documentation can be improved. Also it is not clear if this configuration should be applied to all nodes or only to one master and/or worker and is automatically replicated.

Thank you in advance for the support.

[brandond]

It looks like you're using europe-docker.pkg.dev as the registry, so you'd want to add an entry to the config section with credentials for europe-docker.pkg.dev. This is covered at https://docs.rke2.io/install/containerd_registry_configuration#configs

Registries.yaml is not a cluster resource. The configuration is local to each node, so you must place this configuration on every node that you want to be able to authenticate to your registry. This is also covered at the top of the above-linked page.

[alex-ioma]

By only using the auth field in registries.yaml might actually work. However, I'm not able to validate as I get te following errors:

crictl pull europe-docker.pgk.dev/some-path/image:v0.1.0-alpha

WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
E0604 23:20:24.104891  288825 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory\"" image="europe-docker.pgk.dev/some-path/image:v0.1.0-alpha"
FATA[0000] pulling image: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial unix /var/run/dockershim.sock: connect: no such file or directory"

This suggests might also be related on how containerd has been setup. FYI: I performed a rather default installation of RKE2 (1.28) on Rocky9 via RPM packages.

Could you please provide some assistance on how to circumnavigate this issue so that I can test the registries config?

As mentioned RKE2 is currently working without apparente issues and is able to pull images by injecting the .dockerconfig as k8s secret. There must be something I'm missing here.

Thanks in advance.

[brandond]

It sounds like you've not set up your environment properly to talk to RKE2's containerd socket when using crictl? This is also covered in the docs.
https://docs.rke2.io/reference/cli_tools#containerd

export CRI_CONFIG_FILE=/var/lib/rancher/rke2/agent/etc/crictl.yaml
/var/lib/rancher/rke2/bin/crictl ps

[alex-ioma]

Thank you very much - definitely flew through that. Finally getting closer but still not working - error is:

E0605 00:02:27.456638  357011 remote_image.go:171] "PullImage from image service failed" err="rpc error: code = Unknown desc = failed to pull and unpack image \"europe-docker.pkg.dev/repo/image:v0.1.0-alpha\": failed to resolve reference \"europe-docker.pkg.dev[...]": failed to authorize: failed to fetch oauth token: unexpected status from GET request to https://europe-docker.pkg.dev/v2/token?scope=repository%3A....: 401 Unauthorized

It seems that ctr is able to use the key provided through registries.yaml file, however, it is not able to use all of it's properties such as the fact that token should be obtain at the token_uri.

For example:

{
  ...
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/registry-ro%40cm-controls.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Are you aware of any issue on containerd side about this. Any suggestion on how to troubleshoot.

Best regards.

[brandond]

I would probably go take a look at the upstream docs, and open an issue if necessary.

[alex-ioma]

I've tested different configurations also by interacting directly with config.toml.tmpl within containerd. Issue remains.
I've open a ticket on the source. Will keep you posted.