RKE2 Installation issues using private registry (pulling images, failed to parse system-default-registry, etc.)

[hmonsalv]

Hi, I am installing RKE2 on a single node with Rancher using a private registry and I was experiencing some issues during the process that I would like to share to get some support.

Scenario

Description

Install RKE2 on Ubuntu via Rancher using a private registry (Harbor)

Components:

Rancher v2.6.8
RKE2 v1.23.10+rke2r1
Harbor v2.4.2-ef2e2e56

Rancher configuration: (relevant for the issues detailed)

system-default-registry = registry.example.com/docker.io

Based on #768, my understanding is that we should not use any rewriting/prefixing with the system-default-registry. The reason why we have it, is that this Rancher instance has been used until now for RKE1. And now we are planning to migrate to RKE2.

Issues

1. Image pulling issue

First I created the cluster in Rancher UI, executed the bootstrap command provided there on the server, and it was failing to install initially the rancher-system-agent.service, because it was not able to pull the image registry.example.com/rancher/system-agent-installer-rke2:v1.23.10-rke2r1.

To solve this issue, following rancher/rancher#37441 I created this mirror rewrite:

mirrors:
  registry.example.com:
    endpoint:
    - https://registry.example.com
    rewrite:
      "^rancher/(.*)": "docker.io/rancher/$1"

Questionmark

In rke2.io documentation it is explained that in order to connect to private registries and use them to pull private images on each node, it is needed to create the file /etc/rancher/rke2/registries.yaml with the repositories and mirrors. So I initially created this file on the server before executing the bootstrap command provided by Rancher, but after executing the bootstrap command this file was being overwritten and emptied to:

{"configs":{},"mirrors":null}

I assumed that it was because in the cluster yaml created in Rancher, these values were empty (I didn't add these repositories values there), and maybe they were overwriting the file I manually saved in the file system. I set these registries/mirrors/rewrite values editing the yaml of the cluster creation in Rancher, and it solved the issue. The file /etc/rancher/rke2/registries.yaml was created automatically with the proper content in the file system:

{"configs":{},"mirrors":{"registry.example.com":{"endpoint":["https://registry.example.com"],"rewrite":{"^rancher/(.*)":"rancher/rancher/$1"}}}

But.. is that the expected behavior? At least from the rke.io documentation referenced above is confusing.
I expected that if the file exists in the fs, Rancher was not overwriting it.

Important note (relevant for next issue detailed down below)
I explicitly set system-default-registry value in rancher editing the yaml for the cluster creation as follows (without /docker.io): .spec.rkeConfig.machineSelectorConfig.config.system-default-registry: registry.example.com

2. rke2-server not installing

Once the previous issue was resolved, the rancher-system-agent.service was installed, and this file /etc/rancher/rke2/config.yaml.d/50-rancher.yaml was created on the file system with this value:

{
  ...
  "system-default-registry": "registry.example.com/docker.io",
  ...
}

Why is that?
I explicitly defined in the cluster yaml file on Rancher UI the value .spec.rkeConfig.machineSelectorConfig.config.system-default-registry: registry.example.com (without /docker.io). My expectation here is that this is respected, but it seems that this /etc/rancher/rke2/config.yaml.d/50-rancher.yaml file was created (by rancher-system-agent?) taking the Rancher global setting instead of what I defined in the cluster yaml.

Then the second issue kicked on, and checking the rke2-server.service logs I could see as explained in #768 this error:

Oct 11 09:34:26 ip-10-99-0-231 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
...
Oct 11 09:34:26 ip-10-99-0-231 rke2[2969]: time="2022-10-11T09:34:26Z" level=fatal msg="failed to parse system-default-registry: registries must be valid RFC 3986 URI authorities: registry.example.com/docker.io"
...
Oct 11 09:34:26 ip-10-99-0-231 systemd[1]: Failed to start Rancher Kubernetes Engine v2 (server).

To solve the issue, I edited manually the file /etc/rancher/rke2/config.yaml.d/50-rancher.yaml, removing the /docker.io:

{
  ...
  "system-default-registry": "registry.example.com",
  ...
}

And in the next iteration of the service restart, rke2-server completed to install properly, installing all components defined in cluster yaml defined in Rancher.

Questionmark

it seems like /etc/rancher/rke2/config.yaml.d/50-rancher.yaml is a file Rancher uses to synchronize the cluster, but I was not able to find too much info about how it works.

Once the RKE2 cluster was installed, up and running, out of curiosity I change on Rancher's Global Settings the system-default-registry removing the suffix /docker.io, and then added it again, back to:

system-default-registry = registry.example.com/docker.io

And then the RKE2 cluster stopped working. I logged into the server, and realized that rke2-server.service was stopped.
I checked the file /etc/rancher/rke2/config.yaml.d/50-rancher.yaml and... voila! somehow, this file was "synchronized" with the value I had just changed in the Rancher global setting, and again had this content:

{
  ...
  "system-default-registry": "registry.example.com/docker.io",
  ...
}

I edited again the file manually in order to remove /docker.io, and rke2-server.service started again properly.
And it's running fine until now (few hours).

So my guess is that when a global setting is changed in Rancher it is pushed to the RKE2 clusters into this file /etc/rancher/rke2/config.yaml.d/50-rancher.yaml, but this file is not actively synchronizing with Rancher's settings. Am I right and is this how it works?

If rke2 services on the cluster actively connect to Rancher and pull the actual global settings into /etc/rancher/rke2/config.yaml.d/50-rancher.yaml, this would be very inconvenient as then rke2-server.service on the cluster would fail again and bring the cluster down.

Would appreciate if anyone can clarify to me this doubts.
Thanks in advance for your support.

Kind regards!

[brandond]

If this was an Issue I would transfer it to rancher/rancher, since it sounds like most of your questions are around managing the configuration of Rancher-provisioned clusters, not about RKE2 itself.

I will say that if you are using Rancher to provision your clusters, you should do all of your cluster configuration in Rancher. If you change the content of configuration files managed by the agent, it will be restored by the agent. If you edit the registry configuration, it will be restored by the agent.

You should make your changes to the default registry and registry mirror/rewrites on the Rancher side, rather than trying to correct it by hand on the nodes afterwards.

[hmonsalv]

Thanks @brandond , could you please transfer the discussion to rancher/rancher?

For me the unexpected behavior on Rancher side is:

I explicitly defined in the cluster yaml file on Rancher UI the value .spec.rkeConfig.machineSelectorConfig.config.system-default-registry: registry.example.com (without /docker.io). My expectation here is that this is respected, but it seems that this /etc/rancher/rke2/config.yaml.d/50-rancher.yaml file was created (by rancher-system-agent?) taking the Rancher system-default-registry: global setting (registry.example.com/docker.io) instead of what I defined in the cluster yaml.

[hmonsalv]

Just saw this is a confirmed bug, and fixed on rancher/rancher v2.6.9-rc3.
rancher/rancher#39104

Thanks a lot for your support!

[brandond]

I can't transfer discussions, only issues. Glad to see you found an existing issue on the Rancher side for this.