Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.30] - rke2 ingress upgrade fails 1.31.4 -> 1.31.5 #7660

Closed
brandond opened this issue Feb 4, 2025 · 1 comment
Closed

[Release-1.30] - rke2 ingress upgrade fails 1.31.4 -> 1.31.5 #7660

brandond opened this issue Feb 4, 2025 · 1 comment
Assignees
@brandond @VestigeJ
Milestone
v1.30.10+rke2r1

Comments

@brandond
Copy link
Member

brandond commented Feb 4, 2025

Backport fix for rke2 ingress upgrade fails 1.31.4 -> 1.31.5
@brandond brandond self-assigned this Feb 4, 2025
@brandond brandond added this to the v1.30.10+rke2r1 milestone Feb 11, 2025
@brandond brandond mentioned this issue Feb 11, 2025
Merged
@VestigeJ
Copy link
Contributor

I do not see this issue replicating the way it should be currently on v1.31.4+rke2r1

$ sudo INSTALL_RKE2_VERSION=v1.31.4+rke2r1 INSTALL_RKE2_EXEC=server ./install-rke2.sh

[INFO]  using v1.31.4+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.31.4+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.31.4+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local

$ go_rke2

Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.

$ kgn

NAME              STATUS     ROLES                       AGE   VERSION
ip                NotReady   control-plane,etcd,master   9s    v1.31.4+rke2r1

$ sudo vim /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml
$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-ip                             1/1     Running     0          3m5s
kube-system   etcd-ip                                                 1/1     Running     0          2m42s
kube-system   helm-install-rke2-canal-2gw8q                           0/1     Completed   0          2m58s
kube-system   helm-install-rke2-coredns-jdzdh                         0/1     Completed   0          2m58s
kube-system   helm-install-rke2-ingress-nginx-gvpwx                   0/1     Completed   0          2m58s
kube-system   helm-install-rke2-metrics-server-gqmpt                  0/1     Completed   0          2m58s
kube-system   helm-install-rke2-snapshot-controller-crd-l856k         0/1     Completed   0          2m58s
kube-system   helm-install-rke2-snapshot-controller-z499h             0/1     Completed   1          2m58s
kube-system   helm-install-rke2-snapshot-validation-webhook-9k9zw     0/1     Completed   0          2m58s
kube-system   kube-apiserver-ip                                       1/1     Running     0          3m2s
kube-system   kube-controller-manager-ip                              1/1     Running     0          3m5s
kube-system   kube-proxy-ip                                           1/1     Running     0          3m1s
kube-system   kube-scheduler-ip                                       1/1     Running     0          3m5s
kube-system   rke2-canal-842vm                                        2/2     Running     0          2m49s
kube-system   rke2-coredns-rke2-coredns-55bdf87668-rtqr7              1/1     Running     0          2m50s
kube-system   rke2-coredns-rke2-coredns-autoscaler-65c8c6bd64-s62cp   1/1     Running     0          2m50s
kube-system   rke2-ingress-nginx-controller-cxjqn                     1/1     Running     0          116s
kube-system   rke2-metrics-server-7c85d458bd-ljjkn                    1/1     Running     0          2m15s
kube-system   rke2-snapshot-controller-65bc6fbd57-qs5bj               1/1     Running     0          2m11s
kube-system   rke2-snapshot-validation-webhook-859c7896df-pjt65       1/1     Running     0          2m14s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-ip                             1/1     Running     0          3m9s
kube-system   etcd-ip                                                 1/1     Running     0          2m46s
kube-system   helm-install-rke2-canal-2gw8q                           0/1     Completed   0          3m2s
kube-system   helm-install-rke2-coredns-jdzdh                         0/1     Completed   0          3m2s
kube-system   helm-install-rke2-ingress-nginx-8n96t                   1/1     Running     0          3s
kube-system   helm-install-rke2-metrics-server-gqmpt                  0/1     Completed   0          3m2s
kube-system   helm-install-rke2-snapshot-controller-crd-l856k         0/1     Completed   0          3m2s
kube-system   helm-install-rke2-snapshot-controller-z499h             0/1     Completed   1          3m2s
kube-system   helm-install-rke2-snapshot-validation-webhook-9k9zw     0/1     Completed   0          3m2s
kube-system   kube-apiserver-ip                                       1/1     Running     0          3m6s
kube-system   kube-controller-manager-ip                              1/1     Running     0          3m9s
kube-system   kube-proxy-ip                                           1/1     Running     0          3m5s
kube-system   kube-scheduler-ip                                       1/1     Running     0          3m9s
kube-system   rke2-canal-842vm                                        2/2     Running     0          2m53s
kube-system   rke2-coredns-rke2-coredns-55bdf87668-rtqr7              1/1     Running     0          2m54s
kube-system   rke2-coredns-rke2-coredns-autoscaler-65c8c6bd64-s62cp   1/1     Running     0          2m54s
kube-system   rke2-ingress-nginx-controller-cxjqn                     1/1     Running     0          2m
kube-system   rke2-metrics-server-7c85d458bd-ljjkn                    1/1     Running     0          2m19s
kube-system   rke2-snapshot-controller-65bc6fbd57-qs5bj               1/1     Running     0          2m15s
kube-system   rke2-snapshot-validation-webhook-859c7896df-pjt65       1/1     Running     0          2m18s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS              RESTARTS   AGE
kube-system   cloud-controller-manager-ip                             1/1     Running             0          3m17s
kube-system   etcd-ip                                                 1/1     Running             0          2m54s
kube-system   helm-install-rke2-canal-2gw8q                           0/1     Completed           0          3m10s
kube-system   helm-install-rke2-coredns-jdzdh                         0/1     Completed           0          3m10s
kube-system   helm-install-rke2-ingress-nginx-8n96t                   1/1     Running             0          11s
kube-system   helm-install-rke2-metrics-server-gqmpt                  0/1     Completed           0          3m10s
kube-system   helm-install-rke2-snapshot-controller-crd-l856k         0/1     Completed           0          3m10s
kube-system   helm-install-rke2-snapshot-controller-z499h             0/1     Completed           1          3m10s
kube-system   helm-install-rke2-snapshot-validation-webhook-9k9zw     0/1     Completed           0          3m10s
kube-system   kube-apiserver-ip                                       1/1     Running             0          3m14s
kube-system   kube-controller-manager-ip                              1/1     Running             0          3m17s
kube-system   kube-proxy-ip                                           1/1     Running             0          3m13s
kube-system   kube-scheduler-ip                                       1/1     Running             0          3m17s
kube-system   rke2-canal-842vm                                        2/2     Running             0          3m1s
kube-system   rke2-coredns-rke2-coredns-55bdf87668-rtqr7              1/1     Running             0          3m2s
kube-system   rke2-coredns-rke2-coredns-autoscaler-65c8c6bd64-s62cp   1/1     Running             0          3m2s
kube-system   rke2-ingress-nginx-admission-patch-fqwgf                0/1     ContainerCreating   0          0s
kube-system   rke2-ingress-nginx-controller-cxjqn                     1/1     Terminating         0          2m8s
kube-system   rke2-ingress-nginx-defaultbackend-5d948499f5-gtl5t      0/1     ContainerCreating   0          2s
kube-system   rke2-metrics-server-7c85d458bd-ljjkn                    1/1     Running             0          2m27s
kube-system   rke2-snapshot-controller-65bc6fbd57-qs5bj               1/1     Running             0          2m23s
kube-system   rke2-snapshot-validation-webhook-859c7896df-pjt65       1/1     Running             0          2m26s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS              RESTARTS   AGE
kube-system   cloud-controller-manager-ip                             1/1     Running             0          3m24s
kube-system   etcd-ip                                                 1/1     Running             0          3m1s
kube-system   helm-install-rke2-canal-2gw8q                           0/1     Completed           0          3m17s
kube-system   helm-install-rke2-coredns-jdzdh                         0/1     Completed           0          3m17s
kube-system   helm-install-rke2-ingress-nginx-8n96t                   0/1     Completed           0          18s
kube-system   helm-install-rke2-metrics-server-gqmpt                  0/1     Completed           0          3m17s
kube-system   helm-install-rke2-snapshot-controller-crd-l856k         0/1     Completed           0          3m17s
kube-system   helm-install-rke2-snapshot-controller-z499h             0/1     Completed           1          3m17s
kube-system   helm-install-rke2-snapshot-validation-webhook-9k9zw     0/1     Completed           0          3m17s
kube-system   kube-apiserver-ip                                       1/1     Running             0          3m21s
kube-system   kube-controller-manager-ip                              1/1     Running             0          3m24s
kube-system   kube-proxy-ip                                           1/1     Running             0          3m20s
kube-system   kube-scheduler-ip                                       1/1     Running             0          3m24s
kube-system   rke2-canal-842vm                                        2/2     Running             0          3m8s
kube-system   rke2-coredns-rke2-coredns-55bdf87668-rtqr7              1/1     Running             0          3m9s
kube-system   rke2-coredns-rke2-coredns-autoscaler-65c8c6bd64-s62cp   1/1     Running             0          3m9s
kube-system   rke2-ingress-nginx-controller-cxjqn                     1/1     Terminating         0          2m15s
kube-system   rke2-ingress-nginx-defaultbackend-5d948499f5-gtl5t      0/1     ContainerCreating   0          9s
kube-system   rke2-metrics-server-7c85d458bd-ljjkn                    1/1     Running             0          2m34s
kube-system   rke2-snapshot-controller-65bc6fbd57-qs5bj               1/1     Running             0          2m30s
kube-system   rke2-snapshot-validation-webhook-859c7896df-pjt65       1/1     Running             0          2m33s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-ip                             1/1     Running     0          14m
kube-system   etcd-ip                                                 1/1     Running     0          14m
kube-system   helm-install-rke2-canal-2gw8q                           0/1     Completed   0          14m
kube-system   helm-install-rke2-coredns-jdzdh                         0/1     Completed   0          14m
kube-system   helm-install-rke2-ingress-nginx-8n96t                   0/1     Completed   0          11m
kube-system   helm-install-rke2-metrics-server-gqmpt                  0/1     Completed   0          14m
kube-system   helm-install-rke2-snapshot-controller-crd-l856k         0/1     Completed   0          14m
kube-system   helm-install-rke2-snapshot-controller-z499h             0/1     Completed   1          14m
kube-system   helm-install-rke2-snapshot-validation-webhook-9k9zw     0/1     Completed   0          14m
kube-system   kube-apiserver-ip                                       1/1     Running     0          14m
kube-system   kube-controller-manager-ip                              1/1     Running     0          14m
kube-system   kube-proxy-ip                                           1/1     Running     0          14m
kube-system   kube-scheduler-ip                                       1/1     Running     0          14m
kube-system   rke2-canal-842vm                                        2/2     Running     0          14m
kube-system   rke2-coredns-rke2-coredns-55bdf87668-rtqr7              1/1     Running     0          14m
kube-system   rke2-coredns-rke2-coredns-autoscaler-65c8c6bd64-s62cp   1/1     Running     0          14m
kube-system   rke2-ingress-nginx-controller-fxs5r                     1/1     Running     0          11m
kube-system   rke2-ingress-nginx-defaultbackend-5d948499f5-gtl5t      1/1     Running     0          11m
kube-system   rke2-metrics-server-7c85d458bd-ljjkn                    1/1     Running     0          13m
kube-system   rke2-snapshot-controller-65bc6fbd57-qs5bj               1/1     Running     0          13m
kube-system   rke2-snapshot-validation-webhook-859c7896df-pjt65       1/1     Running     0          13m

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   cloud-controller-manager-ip                             1/1     Running     0          16m
kube-system   etcd-ip                                                 1/1     Running     0          15m
kube-system   helm-install-rke2-canal-2gw8q                           0/1     Completed   0          15m
kube-system   helm-install-rke2-coredns-jdzdh                         0/1     Completed   0          15m
kube-system   helm-install-rke2-ingress-nginx-8n96t                   0/1     Completed   0          12m
kube-system   helm-install-rke2-metrics-server-gqmpt                  0/1     Completed   0          15m
kube-system   helm-install-rke2-snapshot-controller-crd-l856k         0/1     Completed   0          15m
kube-system   helm-install-rke2-snapshot-controller-z499h             0/1     Completed   1          15m
kube-system   helm-install-rke2-snapshot-validation-webhook-9k9zw     0/1     Completed   0          15m
kube-system   kube-apiserver-ip                                       1/1     Running     0          16m
kube-system   kube-controller-manager-ip                              1/1     Running     0          16m
kube-system   kube-proxy-ip                                           1/1     Running     0          15m
kube-system   kube-scheduler-ip                                       1/1     Running     0          16m
kube-system   rke2-canal-842vm                                        2/2     Running     0          15m
kube-system   rke2-coredns-rke2-coredns-55bdf87668-rtqr7              1/1     Running     0          15m
kube-system   rke2-coredns-rke2-coredns-autoscaler-65c8c6bd64-s62cp   1/1     Running     0          15m
kube-system   rke2-ingress-nginx-controller-fxs5r                     1/1     Running     0          12m
kube-system   rke2-ingress-nginx-defaultbackend-5d948499f5-gtl5t      1/1     Running     0          12m
kube-system   rke2-metrics-server-7c85d458bd-ljjkn                    1/1     Running     0          15m
kube-system   rke2-snapshot-controller-65bc6fbd57-qs5bj               1/1     Running     0          15m
kube-system   rke2-snapshot-validation-webhook-859c7896df-pjt65       1/1     Running     0          15m

$ sudo cat /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  annotations:
    helm.cattle.io/chart-url: https://rke2-charts.rancher.io/assets/rke2-ingress-nginx/rke2-ingress-nginx-4.10.503.tgz
    rke2.cattle.io/inject-cluster-config: "true"
  name: rke2-ingress-nginx
  namespace: kube-system
spec:
  valuesContent: |-
    defaultBackend:
      enabled: true
      autoscaling:
        enabled: false
      image:
        repository: cerit.io/cerit/default-backend
        tag: "v1.1"
        readOnlyRootFilesystem: false
  bootstrap: false
  chartContent:  ........
...............
...............
...............
  set:
    global.clusterCIDR: 10.42.0.0/16
    global.clusterCIDRv4: 10.42.0.0/16
    global.clusterDNS: 10.43.0.10
    global.clusterDomain: cluster.local
    global.rke2DataDir: /var/lib/rancher/rke2
    global.serviceCIDR: 10.43.0.0/16
    global.systemDefaultIngressClass: ingress-nginx

I'm truncating the output on these commands to focus on the pods in question from the clusters kube-system namespace FYI

$ sudo INSTALL_RKE2_VERSION=v1.31.6-rc1+rke2r1 INSTALL_RKE2_EXEC=server ./install-rke2.sh

[INFO]  using v1.31.6-rc1+rke2r1 as release
[INFO]  downloading checksums at https://github.com/rancher/rke2/releases/download/v1.31.6-rc1+rke2r1/sha256sum-amd64.txt
[INFO]  downloading tarball at https://github.com/rancher/rke2/releases/download/v1.31.6-rc1+rke2r1/rke2.linux-amd64.tar.gz
[INFO]  verifying tarball
[INFO]  unpacking tarball file to /usr/local

$ go_rke2

Created symlink /etc/systemd/system/multi-user.target.wants/rke2-server.service → /usr/local/lib/systemd/system/rke2-server.service.

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-6qnjw                   0/1     Completed   0          12m
kube-system   rke2-ingress-nginx-controller-q4k6t                     1/1     Running     0          11m

$ sudo vim /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml
$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-6qnjw                   0/1     Completed   0          15m
kube-system   rke2-ingress-nginx-controller-q4k6t                     1/1     Running     0          14m

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-6qnjw                   0/1     Completed   0          15m
kube-system   rke2-ingress-nginx-controller-q4k6t                     1/1     Running     0          14m

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-6qnjw                   0/1     Completed   0          15m
kube-system   rke2-ingress-nginx-controller-q4k6t                     1/1     Running     0          14m

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   1/1     Running     0          3s
kube-system   rke2-ingress-nginx-controller-q4k6t                     1/1     Running     0          14m

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS              RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   1/1     Running             0          13s
kube-system   rke2-ingress-nginx-admission-patch-6fh2r                0/1     Completed           0          3s
kube-system   rke2-ingress-nginx-controller-q4k6t                     1/1     Terminating         0          14m
kube-system   rke2-ingress-nginx-defaultbackend-68d99f5984-cb688      0/1     ContainerCreating   0          4s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS        RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   0/1     Completed     0          20s
kube-system   rke2-ingress-nginx-controller-q4k6t                     1/1     Terminating   0          15m
kube-system   rke2-ingress-nginx-defaultbackend-68d99f5984-cb688      1/1     Running       0          11s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   0/1     Completed   0          24s
kube-system   rke2-ingress-nginx-controller-tzbs8                     0/1     Running     0          3s
kube-system   rke2-ingress-nginx-defaultbackend-68d99f5984-cb688      1/1     Running     0          15s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   0/1     Completed   0          28s
kube-system   rke2-ingress-nginx-controller-tzbs8                     0/1     Running     0          7s
kube-system   rke2-ingress-nginx-defaultbackend-68d99f5984-cb688      1/1     Running     0          19s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   0/1     Completed   0          30s
kube-system   rke2-ingress-nginx-controller-tzbs8                     0/1     Running     0          9s
kube-system   rke2-ingress-nginx-defaultbackend-68d99f5984-cb688      1/1     Running     0          21s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   0/1     Completed   0          41s
kube-system   rke2-ingress-nginx-controller-tzbs8                     0/1     Running     0          20s
kube-system   rke2-ingress-nginx-defaultbackend-68d99f5984-cb688      1/1     Running     0          32s

$ kgp -A

NAMESPACE     NAME                                                    READY   STATUS      RESTARTS   AGE
kube-system   helm-install-rke2-ingress-nginx-2c6kt                   0/1     Completed   0          44s
kube-system   rke2-ingress-nginx-controller-tzbs8                     1/1     Running     0          23s
kube-system   rke2-ingress-nginx-defaultbackend-68d99f5984-cb688      1/1     Running     0          35s

@VestigeJ VestigeJ mentioned this issue Feb 20, 2025
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@VestigeJ VestigeJ

Labels
None yet
Projects
None yet
Milestone
v1.30.10+rke2r1
Development

No branches or pull requests
2 participants
@brandond @VestigeJ