Upgrade RKE2

[cloudcafetech]

Is there any impact on existing cluster when we do RKE2 from one version to another version?

Do I need ETCD restoration after upgradation?

[brandond]

Just the control-plane downtime that comes from rolling restarts of the kubelet and static pods.

No, you do not need to reset or restore the etcd cluster. Can I ask why you thought you might?

[brandond]

It should go to running once etcd and the apiserver are up. If you don't see that happening, check the static pod logs.

[cloudcafetech]

Look below ...

rke2-agent.service - Rancher Kubernetes Engine v2 (agent)
Loaded: loaded (/usr/lib/systemd/system/rke2-agent.service; disabled; vendor preset: disabled)
Active: activating (start) since Fri 2021-10-29 21:38:53 CDT; 2min 17s ago
Docs: https://github.com/rancher/rke2#readme
Process: 3055632 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 3055630 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 3055627 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, >
Main PID: 3055635 (rke2)
Tasks: 8
Memory: 40.8M
CGroup: /system.slice/rke2-agent.service
└─3055635 /usr/bin/rke2 agent

Oct 29 21:40:25 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:40:25-05:00" level=info msg="Waiting to retr>
Oct 29 21:40:30 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:40:30-05:00" level=info msg="Waiting to retr>
Oct 29 21:40:35 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:40:35-05:00" level=info msg="Waiting to retr>
Oct 29 21:40:40 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:40:40-05:00" level=info msg="Waiting to retr>
Oct 29 21:40:45 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:40:45-05:00" level=info msg="Waiting to retr>
Oct 29 21:40:50 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:40:50-05:00" level=info msg="Waiting to retr>
Oct 29 21:40:55 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:40:55-05:00" level=info msg="Waiting to retr>
Oct 29 21:41:00 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:41:00-05:00" level=info msg="Waiting to retr>
Oct 29 21:41:05 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:41:05-05:00" level=info msg="Waiting to retr>
Oct 29 21:41:10 PRANCHERIN01.wkctls.local rke2[3055635]: time="2021-10-29T21:41:10-05:0

• config

server: https://172.25.32.171:9345
token: rancher-secret
node-label:
- "region=worker"
# Run Kubelet with read-only-port 10255
kubelet-arg:
 - "read-only-port=10255"

• REPO

[rancher-rke2-common-stable]
name=Rancher RKE2 Common (v1.21)
baseurl=https://rpm.rancher.io/rke2/stable/common/centos/8/noarch
enabled=1
gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key
[rancher-rke2-1.21-stable]
name=Rancher RKE2 1.21 (v1.21)
baseurl=https://rpm.rancher.io/rke2/stable/1.21/centos/8/x86_64
enabled=1
gpgcheck=1
gpgkey=https://rpm.rancher.io/public.key

[brandond]

Can you get the actual journald logs and not just unit status?

[brandond]

You might also check the pod status and logs via crictl.

[cloudcafetech]

Can you get the actual journald logs and not just unit status?

 [root@PRANCHERIN01 ~]# journalctl -n 50 --no-pager
-- Logs begin at Mon 2021-10-25 18:26:21 CDT, end at Fri 2021-10-29 22:50:47 CDT. --
Oct 29 22:48:42 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:48:42-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:48:47 PRANCHERIN01.wkctls.local sshd[3071580]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:48:47 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:48:47-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:48:52 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:48:52-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:48:57 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:48:57-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:48:59 PRANCHERIN01.wkctls.local sshd[3071598]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:49:02 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:02-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:02 PRANCHERIN01.wkctls.local sshd[3071610]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:49:05 PRANCHERIN01.wkctls.local sshd[3071611]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:49:07 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:07-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:08 PRANCHERIN01.wkctls.local sshd[3071614]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:49:12 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:12-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:17 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:17-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:22 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:22-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:27 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:27-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:32 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:32-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:37 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:37-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:42 PRANCHERIN01.wkctls.local sshd[3071642]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:49:42 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:42-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:47 PRANCHERIN01.wkctls.local sshd[3071644]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:49:47 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:47-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:52 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:52-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:57 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:49:57-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:49:59 PRANCHERIN01.wkctls.local sshd[3071661]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:50:00 PRANCHERIN01.wkctls.local systemd[1]: Starting recap...
Oct 29 22:50:00 PRANCHERIN01.wkctls.local systemd[1]: Starting system activity accounting tool...
Oct 29 22:50:00 PRANCHERIN01.wkctls.local systemd[1]: sysstat-collect.service: Succeeded.
Oct 29 22:50:00 PRANCHERIN01.wkctls.local systemd[1]: Started system activity accounting tool.
Oct 29 22:50:00 PRANCHERIN01.wkctls.local dbus-daemon[924]: [system] Activating via systemd: service name='org.freedesktop.hostname1' unit='dbus-org.freedesktop.hostname1.service' requested by ':1.25343' (uid=0 pid=3071684 comm="hostnamectl --static " label="system_u:system_r:initrc_t:s0")
Oct 29 22:50:00 PRANCHERIN01.wkctls.local systemd[1]: Starting Hostname Service...
Oct 29 22:50:00 PRANCHERIN01.wkctls.local dbus-daemon[924]: [system] Successfully activated service 'org.freedesktop.hostname1'
Oct 29 22:50:00 PRANCHERIN01.wkctls.local systemd[1]: Started Hostname Service.
Oct 29 22:50:02 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:02-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:02 PRANCHERIN01.wkctls.local sshd[3071740]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:50:05 PRANCHERIN01.wkctls.local sshd[3071750]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:50:07 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:07-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:08 PRANCHERIN01.wkctls.local sshd[3071783]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:50:10 PRANCHERIN01.wkctls.local systemd[1]: recap.service: Succeeded.
Oct 29 22:50:10 PRANCHERIN01.wkctls.local systemd[1]: Started recap.
Oct 29 22:50:12 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:12-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:17 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:17-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:22 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:22-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:27 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:27-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:30 PRANCHERIN01.wkctls.local systemd[1]: systemd-hostnamed.service: Succeeded.
Oct 29 22:50:32 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:32-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:37 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:37-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:42 PRANCHERIN01.wkctls.local sshd[3071852]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:50:42 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:42-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"
Oct 29 22:50:47 PRANCHERIN01.wkctls.local sshd[3071853]: error: kex_exchange_identification: Connection closed by remote host
Oct 29 22:50:47 PRANCHERIN01.wkctls.local rke2[3071085]: time="2021-10-29T22:50:47-05:00" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"

You might also check the pod status and logs via crictl.

Looks like if services not run crictl not enabled .... not able to find crictl tool

[root@PRANCHERIN01 ~]# find / -name crictl
[root@PRANCHERIN01 ~]#
[root@PRANCHERIN01 ~]#