RKE2 Upgrade AND migrate from airgap/tarball to private registry - v1.22 to v1.23

[burlyunixguy]

Hello everyone.. got a question... but first a little background

Upgrading rke2 to later releases and doing so stepwise to adhere to skewing policies. First step is to upgrade from rke2 v.1.22 to rke2 v1.23 and migrating from airgap/tarball to private registry. Also had to specify custom image tags in the rke2/config.yaml file.
Have successfully upgraded the master and worker nodes and for a few test nodes.. removed the tarball and local images/containers to ensure we pull from a private registry. Also disabled unused manifest in the rke2/config.yaml file.
The problem is that none of the supporting helm charts were upgraded (canal, nginx-ingress, metrics ).

How does the rke2 upgrade process retrieve and deploy the new helm charts? I also had to create a helmchartconfig resource to specify a different image tag.

Thanks for any and all help.

[brandond]

How does the rke2 upgrade process retrieve and deploy the new helm charts

Charts for embedded components are baked into the rke2 runtime image, and are upgraded as part of rke2 itself.

If you want to use a private registry you have a couple options, in descending order of preference:

1. Use registry mirrors in registries.yaml to set up a your private registry for docker.io, which is the default registry for system images.
2. Use the --system-default-registry option to switch the registry for system images from docker.io, to your private registry
3. Configure HelmChartConfig resources for all the rke2 components, and manually override the images to point at your private registry.

[burlyunixguy]

Thanks for the info. I have the private registry sorted out and the runtime has been upgraded. I have upgraded all the nodes, but the canal, coredns, nginx-ingress, and metrics components were not upgraded. They are still using the old charts. How does rke2 detect when all nodes have been upgraded and initiate a helm upgrade for the embedded components?

[brandond]

They are still using the old charts

The charts are updated when the first server is upgraded. You should see helm job pods running, attempting to update the charts. If it is failing, the logs should say why.

You can also do kubectl get addon,helmchart -A and see the addons and their corresponding charts.

[burlyunixguy]

okay.. thanks again for the responses and answeres. I finally understand what was happening and the handling of add on components within rke2. When I mentioned that we had to specify custom image tags we also have custom image tag for the klipper helm controller. We append a suffix to the existing tag. With the helm controller tag changed, the helm jobs that launch when starting up rke2 fail because the image for the helm job is not found. I can work around in one two ways.

1. Modify the jobs helm pod with the correct image.
2. update the manifests in /var/lib/rancher/rke2/servers/manifests with to override the helm job image ( spec.jobImage parameter ).

The problem with this work around is that it is not persistent. When rke2 restarts via systemd, the deploy controller overwrites the Addon manifest. I have come across the rke2 config option, helm-job-image, but it is not available until v1.24 and my current release v 1.23. I think this option will solve my issues going forward.

Just for anyone else that comes along and has challenges using custom image tags, here are some added notes.
The Rancher RKE2 documentation actually does an excellent job covering it all, but piecing it all together takes an understanding of the rke2 runtime, rke2 deployment controller, rke2 config parameters, HelmChart, HelmChartConfig, and rke2 component Addons.

1. Set parameters for the respective custom image tags in /etc/rancher/rke2/config.yaml or /etc/rancher/rke2/config.yaml.d/*.conf
   These are described in the image parameters described in the Server Configuraion Reference

Can also find from :

rke2 server --help | grep image
rke2 agent --help | grep image

2. Create HelmChartConfig manifest to deploy Addon helm chart with custom values. - Reference RKE2 - Helm Integration

   /var/lib/rancher/rke2/server/manifests

   • rke2-canal-config.yaml
   • rke2-coredns-config.yaml
   • rke2-metrics-server-config.yaml
   • rek2-nginx-ingress-config.yaml

Like I mentioned earlier, If at version <= v1.23 and there is a custom tag for the helm controller, you have to monkey around with he HelmChart manifests in /var/lib/rancher/rke2/servers/manifests/ AND understand that they will be overwritten with rke2 restarts.

[GuillaumeDorschner]

Hey @burlyunixguy,

I'm trying to set up a fully air-gap (so offline) env. I'm using rancher, but I'm getting some basic issues like:

May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=info msg="[Applyinator] Extracting image 192.168.137.50:8082/rancher/system-agent-installer-rke2:v1.27.13-rke2r1 to directory /var/lib/rancher/agent/work/20240515-120455/ab9cf64bd24c75ebeadc57fa049c9cc92>
May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=info msg="Pulling image 192.168.137.50:8082/rancher/system-agent-installer-rke2:v1.27.13-rke2r1"
May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=warning msg="Ignoring invalid endpoint URL for registry 192.168.137.50:8082: parse \"192.168.137.50:8082\": first path segment in URL cannot contain colon"
May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=warning msg="Failed to get image from endpoint: Get \"https://192.168.137.50:8082/v2/\": http: server gave HTTP response to HTTPS client; Get \"https://192.168.137.50:8082/v2/\": http: server gave HTTP r>
May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=error msg="error while staging: all endpoints failed: Get \"https://192.168.137.50:8082/v2/\": http: server gave HTTP response to HTTPS client; Get \"https://192.168.137.50:8082/v2/\": http: server gave >
May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=error msg="error executing instruction 0: all endpoints failed: Get \"https://192.168.137.50:8082/v2/\": http: server gave HTTP response to HTTPS client; Get \"https://192.168.137.50:8082/v2/\": http: se>
May 15 12:04:55 server2 rancher-system-agent[389864]: time="2024-05-15T12:04:55+02:00" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20240515-120455/ab9cf64bd24c75ebeadc57fa049c9cc927226f94c361ff71f126bf0da31d6ecb_0"

• rancher 2.8.2
• registry is nexus
• don't use tls

Here is my current Docker configuration /etc/docker/daemon.json:

{
  "insecure-registries":[
      "repo.labo.bi:8082",
      "192.168.137.50:8082",
      "repo.labo.bi:8083",
      "192.168.137.50:8083"
  ],
  "registry-mirrors":[
      "http://repo.labo.bi:8082",
      "http://192.168.137.50:8082",
      "http://repo.labo.bi:8083",
      "http://192.168.137.50:8083"
  ]
} 

And I'm pulling Docker images like this:

[root@server1 ~]# docker pull 192.168.137.50:8082/nginx
Using default tag: latest
latest: Pulling from nginx
09f376ebb190: Downloading [==>                                                ]  1.179MB/29.15MB
09f376ebb190: Downloading [===>                                               ]   2.06MB/29.15MB
a11fc495bafd: Downloading [==>                                                ]  2.093MB/41.83MB
999643392fb7: Download complete 
09f376ebb190: Downloading [====>                                              ]  2.645MB/29.15MB
45337c09cd57: Download complete 
de3b062c0af7: Download complete 
a11fc495bafd: Downloading [===>                                               ]   2.93MB/41.83MB
09f376ebb190: Downloading [======>                                            ]  3.522MB/29.15MB

##### or since I have a registry mirror I can do like this

[root@server1 ~]# docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
09f376ebb190: Downloading [=====>                                             ]  2.924MB/29.15MB
a11fc495bafd: Downloading [===>                                               ]   2.93MB/41.83MB
933cc8470577: Download complete 
999643392fb7: Download complete 
971bb7f4fb12: Download complete 
45337c09cd57: Download complete 
de3b062c0af7: Download complete

Could i see the url / confgi you set in the etc/rancher/rke2/registries.yaml? I'm quite stuck I will try rke2 without rancher to see if it works.

[burlyunixguy]

I'm not familiar with deploying rke2 through Rancher.. yet.. I went the other way, deployed rke2 clusters, then stood up a separate rancher cluster for users to interface with.

You can also test pulling images with /var/lib/rancher/rke2/bin/crictl .

Here is a portion of the file (but matches what is found at https://docs.rke2.io/install/containerd_registry_configuration ). The file is really irrelevant because all our deployments are updated with the private registry server.

mirrors:
  docker.io:
    endpoint:
      - "https://privateRegistry.myAirGap.com:8083"
  quay.io:
    endpoint:
      - "https://privateRegistry.myAirGap.com:8083"
  k8s.gcr.io:
    endpoint:
      - "https://privateRegistry.myAirGap.com:8083"
  registry.k8s.io:
    endpoint:
      - "https://privateRegistry.myAirGap.com:8083"