Core DNS: Error: UPGRADE FAILED: chart requires kubeVersion: >= v1.31.4 which is incompatible with Kubernetes v1.30.5+rke2r1

[rajivml]

HI

We are attempting an upgrade from 1.30.5 to 1.31.4 on a multi node environment and we perform RKE2 upgrades sequentially, i.e., node by node. During the upgrade of server-0 to v1.31.4, the CoreDNS pod may get scheduled on a node still running the older RKE2 version (v1.30.5), which impacts the pods on the older nodes. Refer to the CoreDNS Helm upgrade log below.

When performing a minor RKE2 version upgrade, such as 1.30.5 to 1.31.4, does Rancher ensure that the shipped version of Rancher components is backward compatible with at least one minor version of RKE2 ?

core-dns logs

[root@server0 testadmin]# kubectl logs helm-install-rke2-coredns-fdh2t -n kube-system
if [[ ${KUBERNETES_SERVICE_HOST} =~ .*:.* ]]; then
    echo "KUBERNETES_SERVICE_HOST is using IPv6"
    CHART="${CHART//%\{KUBERNETES_API\}%/[${KUBERNETES_SERVICE_HOST}]:${KUBERNETES_SERVICE_PORT}}"
else
    CHART="${CHART//%\{KUBERNETES_API\}%/${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}}"
fi
set +v -x
+ [[ '' == \v\2 ]]
+ shopt -s nullglob
+ [[ -f /config/ca-file.pem ]]
+ [[ -f /tmp/ca-file.pem ]]
+ [[ false == \t\r\u\e ]]
+ [[ false == \t\r\u\e ]]
+ [[ -n '' ]]
+ helm_content_decode
+ set -e
+ ENC_CHART_PATH=/chart/rke2-coredns.tgz.base64
+ CHART_PATH=/tmp/rke2-coredns.tgz
+ [[ ! -f /chart/rke2-coredns.tgz.base64 ]]
+ base64 -d /chart/rke2-coredns.tgz.base64
+ CHART=/tmp/rke2-coredns.tgz
+ set +e
+ [[ install != \d\e\l\e\t\e ]]
+ helm_repo_init
+ grep -q -e 'https\?://'
+ [[ /tmp/rke2-coredns.tgz == stable/* ]]
+ [[ -n '' ]]
+ helm_update install --set-string global.clusterCIDR=10.42.0.0/16 --set-string global.clusterCIDRv4=10.42.0.0/16 --set-string global.clusterDNS=10.43.0.10 --set-string global.clusterDomain=cluster.local --set-string global.rke2DataDir=/var/lib/rancher/rke2 --set-string global.serviceCIDR=10.43.0.0/16 --set-string global.systemDefaultIngressClass=ingress-nginx
++ helm ls --all -f '^rke2-coredns$' --namespace kube-system --output json
++ jq -r '"\(.[0].chart),\(.[0].status)"'
++ tr '[:upper:]' '[:lower:]'
+ LINE=rke2-coredns-1.29.006,deployed
+ IFS=,
+ read -r INSTALLED_VERSION STATUS _
+ VALUES=
+ for VALUES_FILE in /config/*.yaml
+ VALUES=' --values /config/values-10_HelmChartConfig.yaml'
+ [[ install = \d\e\l\e\t\e ]]
+ [[ rke2-coredns-1.29.006 =~ ^(|null)$ ]]
+ [[ deployed =~ ^(pending-install|pending-upgrade|pending-rollback|uninstalling)$ ]]
+ [[ deployed =~ ^deployed$ ]]
+ echo 'Already installed rke2-coredns'
+ helm mapkubeapis rke2-coredns --namespace kube-system
Already installed rke2-coredns
2025/01/28 10:51:11 Release 'rke2-coredns' will be checked for deprecated or removed Kubernetes APIs and will be updated if necessary to supported API versions.
2025/01/28 10:51:11 Get release 'rke2-coredns' latest version.
2025/01/28 10:51:11 Check release 'rke2-coredns' for deprecated or removed APIs...
2025/01/28 10:51:11 Finished checking release 'rke2-coredns' for deprecated or removed APIs.
2025/01/28 10:51:11 Release 'rke2-coredns' has no deprecated or removed APIs.
2025/01/28 10:51:11 Map of release 'rke2-coredns' deprecated or removed APIs to supported versions, completed successfully.
+ echo 'Upgrading helm chart'
+ echo 'Upgrading rke2-coredns'
+ shift 1
Upgrading rke2-coredns
+ helm upgrade --set-string global.clusterCIDR=10.42.0.0/16 --set-string global.clusterCIDRv4=10.42.0.0/16 --set-string global.clusterDNS=10.43.0.10 --set-string global.clusterDomain=cluster.local --set-string global.rke2DataDir=/var/lib/rancher/rke2 --set-string global.serviceCIDR=10.43.0.0/16 --set-string global.systemDefaultIngressClass=ingress-nginx rke2-coredns /tmp/rke2-coredns.tgz --values /config/values-10_HelmChartConfig.yaml
Error: UPGRADE FAILED: chart requires kubeVersion: >= v1.31.4 which is incompatible with Kubernetes v1.30.5+rke2r1
+ exit

[brandond]

This is how our charts work. The charts bundled with a release always require that they be executed on a node that has been upgraded to that release. We ensure that by injecting a kubernetes version dependency at build time. Charts are not guaranteed to successfully install or upgrade until after all control-plane nodes have been upgraded. This is known as eventual consistency and is a normal part of Kubernetes cluster operation.

Finish upgrading all of your control-plane nodes, and the helm chart upgrade operation will retry and successfully complete. If you see helm install jobs failing with this chart requires kubeVersion message for an extended period of time, then you are leaving your cluster only partially upgraded for too long. Complete the upgrade on all server nodes and THEN check to see if components have been upgraded successfully.