Newbie on board RKE2/Rancher #3134

surdrazvan · 2022-07-06T16:27:43Z

surdrazvan
Jul 6, 2022

Hi all, I'm a newbie in the Kubernetes and Rancher stuff and I'm struggle with a Kubernetes+Rancher on premise stuff. What i have done till now is to deploy 2 VM's with RHEL 8.4 (Ootpa) on it, and on the first machine i installed the RKE2 server and also Rancher

kubectl get pod -A
NAMESPACE                   NAME                                                    READY   STATUS      RESTARTS      AGE
cattle-fleet-local-system   fleet-agent-699b5fb945-b6dzg                            1/1     Running     1 (20m ago)   82m
cattle-fleet-system         fleet-controller-784d6fbcd8-q8gbr                       1/1     Running     1 (20m ago)   82m
cattle-fleet-system         gitjob-6b977748fc-6lgfd                                 1/1     Running     1 (20m ago)   82m
cattle-system               rancher-864b9b7d48-pwtmg                                1/1     Running     1 (20m ago)   84m
cattle-system               rancher-webhook-5b65595df9-rz97c                        1/1     Running     1 (20m ago)   82m
cert-manager                cert-manager-67599dcc49-9hzmf                           1/1     Running     1 (20m ago)   100m
cert-manager                cert-manager-cainjector-58c8955c5d-m6ftt                1/1     Running     1 (20m ago)   100m
cert-manager                cert-manager-webhook-d698b4885-nqhld                    1/1     Running     1 (20m ago)   100m
kube-system                 cloud-controller-manager-rhel-001                       1/1     Running     2 (20m ago)   109m
kube-system                 etcd-rhel-001                                           1/1     Running     1 (20m ago)   108m
kube-system                 helm-install-rke2-canal-pcmp5                           0/1     Completed   0             109m
kube-system                 helm-install-rke2-coredns-cccdd                         0/1     Completed   0             109m
kube-system                 helm-install-rke2-ingress-nginx-5fw9j                   0/1     Completed   0             109m
kube-system                 helm-install-rke2-metrics-server-2w9sm                  0/1     Completed   0             109m
kube-system                 kube-apiserver-rhel-001                                 1/1     Running     1 (20m ago)   108m
kube-system                 kube-controller-manager-rhel-001                        1/1     Running     2 (20m ago)   109m
kube-system                 kube-proxy-rhel-001                                     1/1     Running     1 (20m ago)   109m
kube-system                 kube-scheduler-rhel-001                                 1/1     Running     1 (20m ago)   109m
kube-system                 rke2-canal-wxwmv                                        2/2     Running     2 (20m ago)   108m
kube-system                 rke2-coredns-rke2-coredns-547d5499cb-q2n9k              1/1     Running     1 (20m ago)   108m
kube-system                 rke2-coredns-rke2-coredns-autoscaler-65c9bb465d-9wtrs   1/1     Running     1 (20m ago)   108m
kube-system                 rke2-ingress-nginx-controller-tmp45                     1/1     Running     1 (20m ago)   107m
kube-system                 rke2-metrics-server-6564db4569-zqd7w                    1/1     Running     1 (20m ago)   108m

How can be see the Rancher UI is running and the local cluster is Active, but when i want to deploy a new cluster i'm receiving the following stuff:

     Configuring bootstrap node(s) custom-dd6e33b28118: waiting for cluster agent to connect

I have run the command journalctl -u rke2-server on the server, and got this output:

-- Logs begin at Wed 2022-07-06 09:49:49 UTC, end at Wed 2022-07-06 15:32:53 UTC. --
Jul 06 14:33:10 rhel-001 systemd[1]: Starting Rancher Kubernetes Engine v2 (server)...
Jul 06 14:33:10 rhel-001 sh[12130]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service
Jul 06 14:33:10 rhel-001 sh[12130]: Failed to get unit file state for nm-cloud-setup.service: No such file or directory
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=warning msg="not running in CIS mode"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Starting rke2 v1.23.7+rke2r2 (d0c2bd7f1dbd30f5b7bbc2e3c899d2efde979c25)"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Managed etcd cluster initializing"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="generated self-signed CA certificate CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10.656236495 +0000 UTC notAfter=2032-07-03 14:33:10.656236495 +0000>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:admin,O=system:masters signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 U>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:kube-controller-manager signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 >
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:kube-scheduler signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:apiserver,O=system:masters signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +00>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:kube-proxy signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:rke2-controller signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=rke2-cloud-controller-manager signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 U>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="generated self-signed CA certificate CN=rke2-server-ca@1657117990: notBefore=2022-07-06 14:33:10.661195281 +0000 UTC notAfter=2032-07-03 14:33:10.661195281 +0000>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=kube-apiserver signed by CN=rke2-server-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="generated self-signed CA certificate CN=rke2-request-header-ca@1657117990: notBefore=2022-07-06 14:33:10.662452204 +0000 UTC notAfter=2032-07-03 14:33:10.6624522>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:auth-proxy signed by CN=rke2-request-header-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="generated self-signed CA certificate CN=etcd-server-ca@1657117990: notBefore=2022-07-06 14:33:10.663511559 +0000 UTC notAfter=2032-07-03 14:33:10.663511559 +0000>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=etcd-server signed by CN=etcd-server-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=etcd-client signed by CN=etcd-server-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="generated self-signed CA certificate CN=etcd-peer-ca@1657117990: notBefore=2022-07-06 14:33:10.665173734 +0000 UTC notAfter=2032-07-03 14:33:10.665173734 +0000 U>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=etcd-peer signed by CN=etcd-peer-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Starting etcd for new cluster"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Running kube-apiserver --advertise-port=6443 --allow-privileged=true --anonymous-auth=false --api-audiences=https://kubernetes.default.svc.cluster.local,rke2 --a>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Running kube-scheduler --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/scheduler.kubeconfig --authorization-kubeconfig=/var/lib/rancher/rke2/server>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Running kube-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/controller.kubeconfig --authorization-ku>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Running cloud-controller-manager --allocate-node-cidrs=true --authentication-kubeconfig=/var/lib/rancher/rke2/server/cred/cloud-controller.kubeconfig --authoriza>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Node token is available at /var/lib/rancher/rke2/server/token"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="To join node to cluster: rke2 agent -s https://10.16.41.96:9345/ -t ${NODE_TOKEN}"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Wrote kubeconfig /etc/rancher/rke2/rke2.yaml"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Run: rke2 kubectl"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=rke2,O=rke2 signed by CN=rke2-server-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=warning msg="dynamiclistener [::]:9345: no cached certificate available for preload - deferring certificate load until storage initialization or first client request"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=rke2,O=rke2 signed by CN=rke2-server-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Tunnel server egress proxy mode: disabled"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Active TLS secret / (ver=) (count 10): map[[listener.cattle.io/cn-10.16.41.96:10.16.41.96](https://listener.cattle.io/cn-10.16.41.96:10.16.41.96) [listener.cattle.io/cn-10.43.0.1:10.43.0.1](https://listener.cattle.io/cn-10.43.0.1:10.43.0.1) [listener.cattle.io/cn-127.0.0.>](https://listener.cattle.io/cn-127.0.0.%3E)
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Active TLS secret / (ver=) (count 10): map[[listener.cattle.io/cn-10.16.41.96:10.16.41.96](https://listener.cattle.io/cn-10.16.41.96:10.16.41.96) [listener.cattle.io/cn-10.43.0.1:10.43.0.1](https://listener.cattle.io/cn-10.43.0.1:10.43.0.1) [listener.cattle.io/cn-127.0.0.>](https://listener.cattle.io/cn-127.0.0.%3E)
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=rhel-001 signed by CN=rke2-server-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +0000 UTC"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="certificate CN=system:node:rhel-001,O=system:nodes signed by CN=rke2-client-ca@1657117990: notBefore=2022-07-06 14:33:10 +0000 UTC notAfter=2023-07-06 14:33:10 +>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Module overlay was already loaded"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Module br_netfilter was already loaded"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Set sysctl 'net/ipv4/conf/all/forwarding' to 1"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_max' to 131072"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_established' to 86400"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Set sysctl 'net/netfilter/nf_conntrack_tcp_timeout_close_wait' to 3600"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Checking local image archives in /var/lib/rancher/rke2/agent/images for [index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2](https://index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2)"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=warning msg="Failed to load runtime image [index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2](https://index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2) from tarball: no local image available for [index.docker.io/rancher/rke2-runti](https://index.docker.io/rancher/rke2-runti)>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Checking local image archives in /var/lib/rancher/rke2/agent/images for [index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2](https://index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2)"
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=warning msg="Failed to load runtime image [index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2](https://index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2) from tarball: no local image available for [index.docker.io/rancher/rke2-runti](https://index.docker.io/rancher/rke2-runti)>
Jul 06 14:33:10 rhel-001 rke2[12142]: time="2022-07-06T14:33:10Z" level=info msg="Pulling runtime image [index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2](https://index.docker.io/rancher/rke2-runtime:v1.23.7-rke2r2)"
Jul 06 14:33:12 rhel-001 rke2[12142]: time="2022-07-06T14:33:12Z" level=info msg="Creating directory /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin"
Jul 06 14:33:12 rhel-001 rke2[12142]: time="2022-07-06T14:33:12Z" level=info msg="Extracting file bin/containerd to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/containerd"
Jul 06 14:33:12 rhel-001 rke2[12142]: time="2022-07-06T14:33:12Z" level=info msg="Extracting file bin/containerd-shim to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/containerd-shim"
Jul 06 14:33:12 rhel-001 rke2[12142]: time="2022-07-06T14:33:12Z" level=info msg="Extracting file bin/containerd-shim-runc-v1 to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/containerd-shim-runc-v1"
Jul 06 14:33:12 rhel-001 rke2[12142]: time="2022-07-06T14:33:12Z" level=info msg="Extracting file bin/containerd-shim-runc-v2 to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/containerd-shim-runc-v2"
Jul 06 14:33:13 rhel-001 rke2[12142]: time="2022-07-06T14:33:13Z" level=info msg="Extracting file bin/crictl to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/crictl"
Jul 06 14:33:13 rhel-001 rke2[12142]: time="2022-07-06T14:33:13Z" level=info msg="Extracting file bin/ctr to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/ctr"
Jul 06 14:33:13 rhel-001 rke2[12142]: time="2022-07-06T14:33:13Z" level=info msg="Extracting file bin/kubectl to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/kubectl"
Jul 06 14:33:14 rhel-001 rke2[12142]: time="2022-07-06T14:33:14Z" level=info msg="Extracting file bin/kubelet to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/kubelet"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file bin/runc to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/bin/runc"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Creating directory /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/harvester-cloud-provider.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/harvester-cloud-provider.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/harvester-csi-driver.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/harvester-csi-driver.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rancher-vsphere-cpi.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rancher-vsphere-cpi.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rancher-vsphere-csi.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rancher-vsphere-csi.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-calico-crd.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-calico-crd.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-calico.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-calico.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-canal.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-canal.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-cilium.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-cilium.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-coredns.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-coredns.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-ingress-nginx.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-ingress-nginx.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-metrics-server.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-metrics-server.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Extracting file charts/rke2-multus.yaml to /var/lib/rancher/rke2/data/v1.23.7-rke2r2-6cac822fc0ea/charts/rke2-multus.yaml"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-csi-driver.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-cpi.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rancher-vsphere-csi.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-cilium.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-coredns.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-ingress-nginx.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-multus.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/harvester-cloud-provider.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico-crd.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-calico.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-canal.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Updated HelmChart /var/lib/rancher/rke2/server/manifests/rke2-metrics-server.yaml to set cluster configuration values"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Logging containerd to /var/lib/rancher/rke2/agent/containerd/containerd.log"
Jul 06 14:33:15 rhel-001 rke2[12142]: time="2022-07-06T14:33:15Z" level=info msg="Running containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/r>
Jul 06 14:33:16 rhel-001 rke2[12142]: time="2022-07-06T14:33:16Z" level=info msg="Containerd is now running"
Jul 06 14:33:16 rhel-001 rke2[12142]: time="2022-07-06T14:33:16Z" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt"
Jul 06 14:33:16 rhel-001 rke2[12142]: time="2022-07-06T14:33:16Z" level=info msg="Pulling image [index.docker.io/rancher/rke2-cloud-provider:v0.0.3-build20211118](https://index.docker.io/rancher/rke2-cloud-provider:v0.0.3-build20211118)..."
Jul 06 14:33:20 rhel-001 rke2[12142]: {"level":"warn","ts":"2022-07-06T14:33:20.767Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f46c40/127>
Jul 06 14:33:20 rhel-001 rke2[12142]: time="2022-07-06T14:33:20Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jul 06 14:33:22 rhel-001 rke2[12142]: time="2022-07-06T14:33:22Z" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/cloud-controller-manager-image.txt in 5.853452109s"
Jul 06 14:33:22 rhel-001 rke2[12142]: time="2022-07-06T14:33:22Z" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/etcd-image.txt"
Jul 06 14:33:22 rhel-001 rke2[12142]: time="2022-07-06T14:33:22Z" level=info msg="Pulling image [index.docker.io/rancher/hardened-etcd:v3.5.4-k3s1-build20220504](https://index.docker.io/rancher/hardened-etcd:v3.5.4-k3s1-build20220504)..."
Jul 06 14:33:28 rhel-001 rke2[12142]: time="2022-07-06T14:33:28Z" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/etcd-image.txt in 5.909140559s"
Jul 06 14:33:28 rhel-001 rke2[12142]: time="2022-07-06T14:33:28Z" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt"
Jul 06 14:33:28 rhel-001 rke2[12142]: time="2022-07-06T14:33:28Z" level=info msg="Pulling image [index.docker.io/rancher/hardened-kubernetes:v1.23.7-rke2r2-build20220608](https://index.docker.io/rancher/hardened-kubernetes:v1.23.7-rke2r2-build20220608)..."
Jul 06 14:33:35 rhel-001 rke2[12142]: {"level":"warn","ts":"2022-07-06T14:33:35.768Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f46c40/127>
Jul 06 14:33:35 rhel-001 rke2[12142]: time="2022-07-06T14:33:35Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jul 06 14:33:40 rhel-001 rke2[12142]: time="2022-07-06T14:33:40Z" level=info msg="Waiting for etcd server to become available"
Jul 06 14:33:40 rhel-001 rke2[12142]: time="2022-07-06T14:33:40Z" level=info msg="Waiting for API server to become available"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-apiserver-image.txt in 12.993956243s"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-controller-manager-image.txt in 292.812µs"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Pulling images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Imported images from /var/lib/rancher/rke2/agent/images/kube-scheduler-image.txt in 1.459533ms"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Running kubelet --address=[0.0.0.0](https://0.0.0.0/) --alsologtostderr=false --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver>
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Connecting to proxy" url="wss://127.0.0.1:9345/v1-rke2/connect"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Handling backend connection request [rhel-001]"
Jul 06 14:33:41 rhel-001 rke2[12142]: time="2022-07-06T14:33:41Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-conf>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --file-check-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-c>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --sync-frequency has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config->
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ f>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --alsologtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --anonymous-auth has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config->
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --authentication-token-webhook has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/k>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --authorization-mode has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-con>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --cgroup-driver has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-f>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --client-ca-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config->
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --cloud-provider has been deprecated, will be removed in 1.24 or later, in favor of removing cloud provider code from Kubelet.
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --cluster-dns has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-fil>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --cluster-domain has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config->
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --containerd has been deprecated, This is a cadvisor flag that was mistakenly registered with the Kubelet. Due to legacy concerns, it will follow the standard CLI deprecation timeline before being rem>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --eviction-hard has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-f>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --eviction-minimum-reclaim has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubel>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --fail-swap-on has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-fi>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --healthz-bind-address has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-c>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --log-file has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-compo>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --log-file-max-size has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in->
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --logtostderr has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8s-co>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --pod-manifest-path has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-conf>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --read-only-port has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config->
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --resolv-conf has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-fil>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --serialize-image-pulls has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet->
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --stderrthreshold has been deprecated, will be removed in a future release, see https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/2845-deprecate-klog-specific-flags-in-k8>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --tls-cert-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-f>
Jul 06 14:33:41 rhel-001 rke2[12142]: Flag --tls-private-key-file has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-c>
Jul 06 14:33:46 rhel-001 rke2[12142]: time="2022-07-06T14:33:46Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jul 06 14:33:50 rhel-001 rke2[12142]: {"level":"warn","ts":"2022-07-06T14:33:50.769Z","logger":"etcd-client","caller":"[email protected]/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000f46c40/127>
Jul 06 14:33:50 rhel-001 rke2[12142]: time="2022-07-06T14:33:50Z" level=info msg="Failed to test data store connection: context deadline exceeded"
Jul 06 14:33:51 rhel-001 rke2[12142]: time="2022-07-06T14:33:51Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jul 06 14:33:55 rhel-001 rke2[12142]: time="2022-07-06T14:33:55Z" level=info msg="Defragmenting etcd database"
Jul 06 14:33:56 rhel-001 rke2[12142]: time="2022-07-06T14:33:56Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jul 06 14:33:57 rhel-001 rke2[12142]: time="2022-07-06T14:33:57Z" level=info msg="etcd data store connection OK"
Jul 06 14:33:57 rhel-001 rke2[12142]: time="2022-07-06T14:33:57Z" level=info msg="Waiting for API server to become available"
Jul 06 14:33:57 rhel-001 rke2[12142]: time="2022-07-06T14:33:57Z" level=info msg="Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap"
Jul 06 14:34:01 rhel-001 rke2[12142]: time="2022-07-06T14:34:01Z" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: https://127.0.0.1:9345/v1-rke2/readyz: 500 Internal Server Error"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Waiting for cloud-controller-manager privileges to become available"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Kube API server is now running"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="rke2 is up and running"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Watching for delete of rhel-001 Node object"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Applying Cluster Role Bindings"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Applying Pod Security Policies"
Jul 06 14:34:02 rhel-001 rke2[12142]: W0706 14:34:02.525128   12142 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Setting PSP: global-unrestricted-psp"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Applying CRD [addons.k3s.cattle.io](https://addons.k3s.cattle.io/)"
Jul 06 14:34:02 rhel-001 rke2[12142]: W0706 14:34:02.535234   12142 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Setting Cluster Role: global-unrestricted-psp-clusterrole"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Applying CRD [helmcharts.helm.cattle.io](https://helmcharts.helm.cattle.io/)"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Setting Cluster RoleBinding: global-unrestricted-psp-rolebinding"
Jul 06 14:34:02 rhel-001 rke2[12142]: W0706 14:34:02.588859   12142 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Setting PSP: system-unrestricted-psp"
Jul 06 14:34:02 rhel-001 rke2[12142]: W0706 14:34:02.593850   12142 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Applying CRD [helmchartconfigs.helm.cattle.io](https://helmchartconfigs.helm.cattle.io/)"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Setting Cluster Role: system-unrestricted-psp-role"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Setting Cluster RoleBinding: system-unrestricted-node-psp-rolebinding"
Jul 06 14:34:02 rhel-001 rke2[12142]: time="2022-07-06T14:34:02Z" level=info msg="Setting Cluster RoleBinding: system-unrestricted-svc-acct-psp-rolebinding"
Jul 06 14:34:02 rhel-001 rke2[12142]: W0706 14:34:02.646009   12142 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Jul 06 14:34:02 rhel-001 rke2[12142]: W0706 14:34:02.650865   12142 warnings.go:70] policy/v1beta1 PodSecurityPolicy is deprecated in v1.21+, unavailable in v1.25+
Jul 06 14:34:03 rhel-001 rke2[12142]: W0706 14:34:03.130415   12142 reflector.go:324] [k8s.io/[email protected]/tools/cache/reflector.go:167:](https://k8s.io/[email protected]/tools/cache/reflector.go:167:) failed to list *v1.Endpoints: endpoints "kubernetes" is forbidden: User "system:rke2-controller" >
Jul 06 14:34:03 rhel-001 rke2[12142]: E0706 14:34:03.130467   12142 reflector.go:138] [k8s.io/[email protected]/tools/cache/reflector.go:167:](https://k8s.io/[email protected]/tools/cache/reflector.go:167:) Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints "kubernetes" is forbidden:>
Jul 06 14:34:03 rhel-001 rke2[12142]: time="2022-07-06T14:34:03Z" level=info msg="Annotations and labels have been set successfully on node: rhel-001"
Jul 06 14:34:03 rhel-001 systemd[1]: Started Rancher Kubernetes Engine v2 (server).
Jul 06 14:34:03 rhel-001 rke2[12142]: time="2022-07-06T14:34:03Z" level=info msg="Updating annotations on kube-system namespace"
Jul 06 14:34:03 rhel-001 rke2[12142]: time="2022-07-06T14:34:03Z" level=info msg="updating namespace: kube-system"
Jul 06 14:34:03 rhel-001 rke2[12142]: time="2022-07-06T14:34:03Z" level=info msg="Pod Security Policies applied successfully"
Jul 06 14:34:04 rhel-001 rke2[12142]: W0706 14:34:04.004998   12142 reflector.go:324] [k8s.io/[email protected]/tools/cache/reflector.go:167:](https://k8s.io/[email protected]/tools/cache/reflector.go:167:) failed to list *v1.Endpoints: endpoints "kubernetes" is forbidden: User "system:rke2-controller" >
Jul 06 14:34:04 rhel-001 rke2[12142]: E0706 14:34:04.005239   12142 reflector.go:138] [k8s.io/[email protected]/tools/cache/reflector.go:167:](https://k8s.io/[email protected]/tools/cache/reflector.go:167:) Failed to watch *v1.Endpoints: failed to list *v1.Endpoints: endpoints "kubernetes" is forbidden:>
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Cluster Role Bindings applied successfully"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Failed to get existing traefik HelmChart" error="[helmcharts.helm.cattle.io](https://helmcharts.helm.cattle.io/) \"traefik\" not found"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Starting managed etcd node label controller"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=error msg="Failed to record snapshots for cluster: No snapshot configmap found"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Starting [k3s.cattle.io/v1](https://k3s.cattle.io/v1), Kind=Addon controller"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Creating deploy event broadcaster"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"harvester-cloud-provider\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion>
Jul 06 14:34:05 rhel-001 rke2[12142]: I0706 14:34:05.698064   12142 leaderelection.go:248] attempting to acquire leader lease kube-system/rke2...
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Starting /v1, Kind=Node controller"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"harvester-csi-driver\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion:\"\>
Jul 06 14:34:05 rhel-001 rke2[12142]: I0706 14:34:05.710986   12142 leaderelection.go:258] successfully acquired lease kube-system/rke2
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Creating svccontroller event broadcaster"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Creating reencrypt-controller event broadcaster"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Starting managed etcd member removal controller"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Starting managed etcd apiserver addresses controller"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Cluster dns configmap has been set successfully"
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rancher-vsphere-cpi\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion:\"\">
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rancher-vsphere-csi\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion:\"\">
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-calico-crd\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion:\"\", Fi>
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-calico\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion:\"\", FieldP>
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-canal\", UID:\"521b4867-1311-4da8-80ac-5e22f4edca57\", APIVersion:\"[k3s.cattle.io](https://k3s.cattle.io/)>
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-canal\", UID:\"521b4867-1311-4da8-80ac-5e22f4edca57\", APIVersion:\"[k3s.cattle.io](https://k3s.cattle.io/)>
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-cilium\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion:\"\", FieldP>
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1da5cddb-15b8-4a98-9cdf-a5153f6f6666\", APIVersion:\"k3s.cattle.>
Jul 06 14:34:05 rhel-001 rke2[12142]: time="2022-07-06T14:34:05Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1da5cddb-15b8-4a98-9cdf-a5153f6f6666\", APIVersion:\"k3s.cattle.>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"61176f34-c05c-459c-a438-98eafe7ce4c0\", APIVersion:\"k3s.c>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"61176f34-c05c-459c-a438-98eafe7ce4c0\", APIVersion:\"k3s.c>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Starting [helm.cattle.io/v1](https://helm.cattle.io/v1), Kind=HelmChart controller"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Starting [helm.cattle.io/v1](https://helm.cattle.io/v1), Kind=HelmChartConfig controller"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-canal\", UID:\"256d2387-6331-49a8-bfed-f19d743b31ca\", APIVersion:\"helm.catt>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1c68c183-5d00-4bb1-b82e-3e4d9d383a9f\", APIVersion:\"[helm.ca](https://helm.ca/)>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"557bb51e-e7ac-4fa9-a3bd-8491a548ee2b\", APIVersion:\"h>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Starting apps/v1, Kind=DaemonSet controller"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Starting apps/v1, Kind=Deployment controller"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Starting [rbac.authorization.k8s.io/v1](https://rbac.authorization.k8s.io/v1), Kind=ClusterRoleBinding controller"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Starting batch/v1, Kind=Job controller"
Jul 06 14:34:06 rhel-001 rke2[12142]: I0706 14:34:06.754927   12142 request.go:665] Waited for 1.027615138s due to client-side throttling, not priority and fairness, request: PUT:https://127.0.0.1:6443/api/v1/nodes/rhel-001
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"674b6337-8a1b-41c8-bf4e-699aac0b4ba4\", APIVersion:\"k3s.>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"674b6337-8a1b-41c8-bf4e-699aac0b4ba4\", APIVersion:\"k3s.>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"9b11b11a-1f47-439e-ad3b-e578ae8f9999\", APIVersion:\">
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Stopped tunnel to [127.0.0.1:9345](https://127.0.0.1:9345/)"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Connecting to proxy" url="wss://10.16.41.96:9345/v1-rke2/connect"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Proxy done" err="context canceled" url="wss://127.0.0.1:9345/v1-rke2/connect"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="error in remotedialer server [400]: websocket: close 1006 (abnormal closure): unexpected EOF"
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"557bb51e-e7ac-4fa9-a3bd-8491a548ee2b\", APIVersion:\"h>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1c68c183-5d00-4bb1-b82e-3e4d9d383a9f\", APIVersion:\"[helm.ca](https://helm.ca/)>
Jul 06 14:34:06 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-canal\", UID:\"256d2387-6331-49a8-bfed-f19d743b31ca\", APIVersion:\"helm.catt>
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:06Z" level=info msg="Event(v1.ObjectReference{Kind:\"Addon\", Namespace:\"kube-system\", Name:\"rke2-multus\", UID:\"\", APIVersion:\"[k3s.cattle.io/v1\](https://k3s.cattle.io/v1/)", ResourceVersion:\"\", FieldP>
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Handling backend connection request [rhel-001]"
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Running kube-proxy --cluster-cidr=[10.42.0.0/16](https://10.42.0.0/16) --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz->
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"9b11b11a-1f47-439e-ad3b-e578ae8f9999\", APIVersion:\">
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1c68c183-5d00-4bb1-b82e-3e4d9d383a9f\", APIVersion:\"[helm.ca](https://helm.ca/)>
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Starting /v1, Kind=Service controller"
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Starting /v1, Kind=Endpoints controller"
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Starting /v1, Kind=Secret controller"
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Starting /v1, Kind=ConfigMap controller"
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Starting /v1, Kind=ServiceAccount controller"
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Starting /v1, Kind=Pod controller"
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1c68c183-5d00-4bb1-b82e-3e4d9d383a9f\", APIVersion:\"[helm.ca](https://helm.ca/)>
Jul 06 14:34:07 rhel-001 rke2[12142]: time="2022-07-06T14:34:07Z" level=info msg="Labels and annotations have been set successfully on node: rhel-001"
Jul 06 14:34:08 rhel-001 rke2[12142]: time="2022-07-06T14:34:08Z" level=info msg="Updating TLS secret for kube-system/rke2-serving (count: 10): map[[listener.cattle.io/cn-10.16.41.96:10.16.41.96](https://listener.cattle.io/cn-10.16.41.96:10.16.41.96) [listener.cattle.io/cn-10.43.0.1:10.43.0.1](https://listener.cattle.io/cn-10.43.0.1:10.43.0.1) listene>
Jul 06 14:34:08 rhel-001 rke2[12142]: time="2022-07-06T14:34:08Z" level=info msg="Active TLS secret kube-system/rke2-serving (ver=407) (count 10): map[[listener.cattle.io/cn-10.16.41.96:10.16.41.96](https://listener.cattle.io/cn-10.16.41.96:10.16.41.96) [listener.cattle.io/cn-10.43.0.1:10.43.0.1](https://listener.cattle.io/cn-10.43.0.1:10.43.0.1) list>
Jul 06 14:34:18 rhel-001 rke2[12142]: time="2022-07-06T14:34:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1c68c183-5d00-4bb1-b82e-3e4d9d383a9f\", APIVersion:\"[helm.ca](https://helm.ca/)>
Jul 06 14:34:18 rhel-001 rke2[12142]: time="2022-07-06T14:34:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-canal\", UID:\"256d2387-6331-49a8-bfed-f19d743b31ca\", APIVersion:\"helm.catt>
Jul 06 14:34:18 rhel-001 rke2[12142]: time="2022-07-06T14:34:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"9b11b11a-1f47-439e-ad3b-e578ae8f9999\", APIVersion:\">
Jul 06 14:34:18 rhel-001 rke2[12142]: time="2022-07-06T14:34:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"557bb51e-e7ac-4fa9-a3bd-8491a548ee2b\", APIVersion:\"h>
Jul 06 14:34:36 rhel-001 rke2[12142]: time="2022-07-06T14:34:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1c68c183-5d00-4bb1-b82e-3e4d9d383a9f\", APIVersion:\"[helm.ca](https://helm.ca/)>
Jul 06 14:34:36 rhel-001 rke2[12142]: time="2022-07-06T14:34:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-coredns\", UID:\"1c68c183-5d00-4bb1-b82e-3e4d9d383a9f\", APIVersion:\"[helm.ca](https://helm.ca/)>
Jul 06 14:34:36 rhel-001 rke2[12142]: time="2022-07-06T14:34:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-canal\", UID:\"256d2387-6331-49a8-bfed-f19d743b31ca\", APIVersion:\"helm.catt>
Jul 06 14:34:36 rhel-001 rke2[12142]: time="2022-07-06T14:34:36Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-canal\", UID:\"256d2387-6331-49a8-bfed-f19d743b31ca\", APIVersion:\"helm.catt>
Jul 06 14:35:18 rhel-001 rke2[12142]: time="2022-07-06T14:35:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"9b11b11a-1f47-439e-ad3b-e578ae8f9999\", APIVersion:\">
Jul 06 14:35:18 rhel-001 rke2[12142]: time="2022-07-06T14:35:18Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-metrics-server\", UID:\"9b11b11a-1f47-439e-ad3b-e578ae8f9999\", APIVersion:\">
Jul 06 14:35:33 rhel-001 rke2[12142]: time="2022-07-06T14:35:33Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"557bb51e-e7ac-4fa9-a3bd-8491a548ee2b\", APIVersion:\"h>
Jul 06 14:35:33 rhel-001 rke2[12142]: time="2022-07-06T14:35:33Z" level=info msg="Event(v1.ObjectReference{Kind:\"HelmChart\", Namespace:\"kube-system\", Name:\"rke2-ingress-nginx\", UID:\"557bb51e-e7ac-4fa9-a3bd-8491a548ee2b\", APIVersion:\"h>
Jul 06 15:00:17 rhel-001 rke2[12142]: time="2022-07-06T15:00:17Z" level=info msg="Updating TLS secret for kube-system/rke2-serving (count: 11): map[[field.cattle.io/projectId:local:p-l5rv4](https://field.cattle.io/projectId:local:p-l5rv4) [listener.cattle.io/cn-10.16.41.96:10.16.41.96](https://listener.cattle.io/cn-10.16.41.96:10.16.41.96) listener.>
Jul 06 15:00:17 rhel-001 rke2[12142]: time="2022-07-06T15:00:17Z" level=info msg="Active TLS secret kube-system/rke2-serving (ver=6656) (count 11): map[[field.cattle.io/projectId:local:p-l5rv4](https://field.cattle.io/projectId:local:p-l5rv4) [listener.cattle.io/cn-10.16.41.96:10.16.41.96](https://listener.cattle.io/cn-10.16.41.96:10.16.41.96) liste>

and also on the machine where i want to create the new cluster i run the command journalctl -u rancher-system-agent, and received the following output:

-- Logs begin at Wed 2022-07-06 09:51:42 UTC, end at Wed 2022-07-06 15:46:30 UTC. --
Jul 06 15:16:47 rhel-002 systemd[1]: Started Rancher System Agent.
Jul 06 15:16:47 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:47Z" level=info msg="Rancher System Agent version v0.2.7 (65aac83) is starting"
Jul 06 15:16:47 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:47Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Jul 06 15:16:47 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:47Z" level=info msg="Starting remote watch of plans"
Jul 06 15:16:48 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:48Z" level=info msg="Starting /v1, Kind=Secret controller"
Jul 06 15:16:48 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:48Z" level=info msg="Detected first start, force-applying one-time instruction set"
Jul 06 15:16:48 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:48Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0"
Jul 06 15:16:48 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:48Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-rke2:v1.22.10-rke2r2 to directory /var/lib/rancher/agent/work/20220706-151648/579334>
Jul 06 15:16:48 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:48Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Jul 06 15:16:48 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:48Z" level=info msg="Pulling image [index.docker.io/rancher/system-agent-installer-rke2:v1.22.10-rke2r2](https://index.docker.io/rancher/system-agent-installer-rke2:v1.22.10-rke2r2)"
Jul 06 15:16:49 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:49Z" level=info msg="Extracting file [installer.sh](https://installer.sh/) to /var/lib/rancher/agent/work/20220706-151648/57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0/ins>
Jul 06 15:16:49 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:49Z" level=info msg="Extracting file rke2.linux-amd64.tar.gz to /var/lib/rancher/agent/work/20220706-151648/57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb36>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="Extracting file sha256sum-amd64.txt to /var/lib/rancher/agent/work/20220706-151648/57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="Extracting file [run.sh](https://run.sh/) to /var/lib/rancher/agent/work/20220706-151648/57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0/run.sh"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[Applyinator] Running command: sh [-c [run.sh](https://run.sh/)]"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + SA_INSTALL_PREFIX=/usr/local"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + mkdir -p /var/lib/rancher/rke2"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + SAI_FILE_DIR=/var/lib/rancher/rke2/system-agent-installer"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/rke2/system-agent-installer/rke>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + RKE2_SA_ENV_FILE_NAME=rke2-sa.env"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' '!' -d /var/lib/rancher/rke2/system-agent-installer ']'"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + mkdir -p /var/lib/rancher/rke2/system-agent-installer"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + check_target_mountpoint"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + mountpoint -q ''"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + check_target_ro"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + touch /usr/local/.rke2-ro-test"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + rm -rf /usr/local/.rke2-ro-test"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + test 0 -ne 0"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + SYSTEMD_BASE_PATH=/usr/local/lib/systemd/system"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + RKE2_SA_ENV_FILE_PATH=/var/lib/rancher/rke2/system-agent-installer/>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + RKE2_SA_ENV_SRV_REF=EnvironmentFile=-/var/lib/rancher/rke2/system-a>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2_restart_st>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' -n 7d900ba4bd8fe641f6ccbce6e568824a9eaa2789ce96c54152fccf8686c2>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' '' '!=' 7d900ba4bd8fe641f6ccbce6e568824a9eaa2789ce96c54152fccf8>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + RESTART=true"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + env INSTALL_RKE2_ARTIFACT_PATH=/var/lib/rancher/agent/work/20220706>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stdout]: [INFO]  staging local checksums from /var/lib/rancher/agent/work/2022>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stdout]: [INFO]  staging tarball from /var/lib/rancher/agent/work/20220706-151>
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stdout]: [INFO]  verifying tarball"
Jul 06 15:16:50 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:50Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stdout]: [INFO]  unpacking tarball file to /usr/local"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' -f /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + OLD_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + install -m 600 /dev/null /var/lib/rancher/rke2/system-agent-install>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ env"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ grep '^RKE2_'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ true"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + RKE2_ENV="
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' -n '' ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ env"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ grep -Ei '^(NO|HTTP|HTTPS)_PROXY'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ true"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + PROXY_ENV_INFO="
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' -n '' ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ sha256sum /var/lib/rancher/rke2/system-agent-installer/rke2-sa.env"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: ++ awk '{print $1}'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + NEW_ENV_FILE_PATH_HASH=e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b85>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' -z '' ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + INSTALL_RKE2_TYPE=server"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + grep -q EnvironmentFile=-/var/lib/rancher/rke2/system-agent-install>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + echo EnvironmentFile=-/var/lib/rancher/rke2/system-agent-installer/>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' -n 7d900ba4bd8fe641f6ccbce6e568824a9eaa2789ce96c54152fccf8686c2>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + echo 7d900ba4bd8fe641f6ccbce6e568824a9eaa2789ce96c54152fccf8686c227>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + systemctl daemon-reload"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' '' = true ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' server = server ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + systemctl is-active --quiet rke2-agent"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + systemctl enable rke2-server"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/rke2-serv>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' '' = true ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + '[' true = true ']'"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stderr]: + systemctl --no-block restart rke2-server"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[Applyinator] Command sh [-c [run.sh](https://run.sh/)] finished with err: <nil> and exit code: 0"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20220706-151648/57933490cd3d711da567a7701d6464aaf2d0>
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Jul 06 15:16:52 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:52Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 1"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:16:53 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:53Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:16:58 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:58Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:16:58 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:58Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:16:58 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:58Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:16:58 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:58Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:16:58 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:16:58Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:03 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:03Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:17:03 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:03Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:03 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:03Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:03 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:03Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:03 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:03Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:08 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:08Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:17:08 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:08Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:08 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:08Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:08 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:08Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:08 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:08Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:13 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:13Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:17:13 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:13Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:13 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:13Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:13 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:13Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:13 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:13Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:18 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:18Z" level=error msg="error encountered during parsing of last run time: parsing time \"\" as \"Mon Jan _2 15:04:05 MST 2006\": cannot parse \"\" as \"Mon\""
Jul 06 15:17:18 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:18Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20220706-151718/57933490cd3d711da567a7701d6464aaf2d0>
Jul 06 15:17:18 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:18Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stdout]: Name Location Size Created"
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:19Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:24 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:24Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:24 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:24Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:24 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:24Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:24 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:24Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:29 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:29Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:34 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:34Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:34 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:34Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:34 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:34Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:34 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:34Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:39 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:39Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:39 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:39Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:39 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:39Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:39 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:39Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:17:41 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:41Z" level=error msg="error loading CA cert for probe (kube-controller-manager) /var/lib/rancher/rke2/server/tls/kube-controller-manager/kube-controller-manager.crt: o>
Jul 06 15:17:41 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:41Z" level=error msg="error while appending ca cert to pool for probe kube-controller-manager"
Jul 06 15:17:41 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:41Z" level=error msg="error loading CA cert for probe (kube-scheduler) /var/lib/rancher/rke2/server/tls/kube-scheduler/kube-scheduler.crt: open /var/lib/rancher/rke2/s>
Jul 06 15:17:41 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:17:41Z" level=error msg="error while appending ca cert to pool for probe kube-scheduler"
Jul 06 15:27:18 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:27:18Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20220706-152718/57933490cd3d711da567a7701d6464aaf2d0>
Jul 06 15:27:18 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:27:18Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Jul 06 15:27:18 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:27:18Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stdout]: Name Location Size Created"
Jul 06 15:27:18 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:27:18Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
Jul 06 15:37:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:37:19Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20220706-153719/57933490cd3d711da567a7701d6464aaf2d0>
Jul 06 15:37:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:37:19Z" level=info msg="[Applyinator] Running command: sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Jul 06 15:37:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:37:19Z" level=info msg="[57933490cd3d711da567a7701d6464aaf2d0f1ba8c26df372f2379acb368bfa0_0:stdout]: Name Location Size Created"
Jul 06 15:37:19 rhel-002 rancher-system-agent[2058]: time="2022-07-06T15:37:19Z" level=info msg="[Applyinator] Command sh [-c rke2 etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

this is the command provided by Rancher UI to deploy the new cluster nodes

Many thanks in advanced for yours responses!
Razvan

Answered by brandond

Jul 6, 2022

It looks like RKE2 has been installed on the first node of the downstream cluster and is running normally. What do you see in the logs of rancher pods on the downstream cluster?

View full answer

brandond · 2022-07-06T16:58:03Z

brandond
Jul 6, 2022
Maintainer

It looks like RKE2 has been installed on the first node of the downstream cluster and is running normally. What do you see in the logs of rancher pods on the downstream cluster?

23 replies

surdrazvan Jul 7, 2022
Author

it's there, the PTR stuff was from the Revers Lookup Zone

brandond Jul 7, 2022
Maintainer

Can you resolve that with dig or nslookup on the host, now that the hosts file entry is gone?

surdrazvan Jul 7, 2022
Author

i solved the issue :), now my cluster is up and running. What i have done it's to deploy a new VM with RHEL 8.4 on it and installed the dnsmasq on it, i edited the entries there and on all the nodes i pointed the new machine as a DNS server and worked :)

surdrazvan Jul 7, 2022
Author

NAMESPACE NAME READY STATUS RESTARTS AGE
calico-system calico-kube-controllers-5bbd8d8bdf-7z58x 1/1 Running 5 (80m ago) 19h
calico-system calico-node-45pzx 1/1 Running 5 (80m ago) 19h
calico-system calico-typha-6d8c467f56-hw7j5 1/1 Running 5 (80m ago) 19h
cattle-fleet-system fleet-agent-96f6f455c-c4kzv 1/1 Running 0 78m
cattle-system cattle-cluster-agent-56d49858b5-xr9q6 1/1 Running 0 79m
cattle-system system-upgrade-controller-5b6457d644-jrhh6 1/1 Running 0 78m
kube-system etcd-rhel-002 1/1 Running 5 (80m ago) 19h
kube-system helm-install-rancher-vsphere-cpi-qxxzz 0/1 Completed 0 19h
kube-system helm-install-rancher-vsphere-csi-8zbbb 0/1 Completed 0 19h
kube-system helm-install-rke2-calico-crd-tl6vz 0/1 Completed 0 19h
kube-system helm-install-rke2-calico-x7rzc 0/1 Completed 2 19h
kube-system helm-install-rke2-coredns-g5w77 0/1 Completed 0 19h
kube-system helm-install-rke2-ingress-nginx-cpl9x 0/1 Completed 0 19h
kube-system helm-install-rke2-metrics-server-x6jbc 0/1 Completed 0 19h
kube-system kube-apiserver-rhel-002 1/1 Running 5 (80m ago) 19h
kube-system kube-controller-manager-rhel-002 1/1 Running 10 (80m ago) 19h
kube-system kube-proxy-rhel-002 1/1 Running 5 (80m ago) 19h
kube-system kube-scheduler-rhel-002 1/1 Running 5 (80m ago) 19h
kube-system rancher-vsphere-cpi-cloud-controller-manager-dhb59 1/1 Running 5 (80m ago) 19h
kube-system rke2-coredns-rke2-coredns-687554ff58-w2m62 1/1 Running 5 (80m ago) 19h
kube-system rke2-coredns-rke2-coredns-autoscaler-7566b44b85-rrmsf 1/1 Running 5 (80m ago) 19h
kube-system rke2-ingress-nginx-controller-pk9b2 1/1 Running 5 (80m ago) 19h
kube-system rke2-metrics-server-8574659c85-q87ng 1/1 Running 5 (80m ago) 19h
kube-system vsphere-csi-controller-5f48cd7cc6-4lrvl 5/5 Running 32 (79m ago) 19h
kube-system vsphere-csi-controller-5f48cd7cc6-klqj2 5/5 Running 32 (79m ago) 19h
kube-system vsphere-csi-controller-5f48cd7cc6-xlfb2 5/5 Running 31 (79m ago) 19h
kube-system vsphere-csi-node-mpw9r 3/3 Running 15 (80m ago) 19h
tigera-operator tigera-operator-7b5df56f45-rhtdz 1/1 Running 5 (80m ago) 19h

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Newbie on board RKE2/Rancher #3134

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment 23 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Newbie on board RKE2/Rancher #3134

surdrazvan Jul 6, 2022

Replies: 1 comment · 23 replies

brandond Jul 6, 2022 Maintainer

surdrazvan Jul 7, 2022 Author

surdrazvan Jul 7, 2022 Author

brandond Jul 7, 2022 Maintainer

surdrazvan Jul 7, 2022 Author

surdrazvan Jul 7, 2022 Author

surdrazvan
Jul 6, 2022

Replies: 1 comment 23 replies

brandond
Jul 6, 2022
Maintainer

surdrazvan Jul 7, 2022
Author

surdrazvan Jul 7, 2022
Author

brandond Jul 7, 2022
Maintainer

surdrazvan Jul 7, 2022
Author

surdrazvan Jul 7, 2022
Author