Cannot create a RKE2 or K3S cluster

[klauserber]

I have problems to create RKE2 or K3S clusters. RKE clusters are working fine.

Test scenario:

• Create a empty Hetzner Project with a API Key and a private network
• Create a Rancher instance on a CX21 machine according to the Rancher quickstart guide: https://ranchermanager.docs.rancher.com/getting-started/quick-start-guides/deploy-rancher-manager/helm-cli
• Install the Hetzner node driver as here documented (tested with versions 3.3.0 and 3.10.1)
• Workaround the credentials issue: Provision rke2 cluster from 3rd party node driver rancher/rancher#34782 (comment)
• Create a RKE2 cluster (K3s is the same) with default values. Networks: network-1, serverType: cx21

Test results:

• Rancher creates a Hetzner machine
• Stucks on 'waiting for agent to check in and apply initial plan'

Provisioning log:

9:40:07 am | [INFO ] waiting for viable init node
9:40:43 am | [INFO ] configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for agent to check in and apply initial plan

Log Rancher pod:

2023/02/08 08:40:06 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:06 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:06 [ERROR] error syncing 'fleet-default/test-rke2-pool1-43b82d16-sldnj': handler machine-provision: no machine owner ref, requeuing
2023/02/08 08:40:06 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:06 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:07 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:07 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:07 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:07 [INFO] EnsureSecretForServiceAccount: waiting for secret [test-rke2-bootstrap-template-nmzss-machine-bootstrap-token6krcw] to be populated with token
2023/02/08 08:40:07 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:07 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:08 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:09 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:09 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:09 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:09 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:10 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:10 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:10 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:10 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:10 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:16 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: waiting for viable init node
2023/02/08 08:40:41 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for bootstrap etcd to be available
2023/02/08 08:40:41 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:40:41 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:40:41 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:40:42 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for agent to check in and apply initial plan
2023/02/08 08:40:42 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:40:42 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for agent to check in and apply initial plan
2023/02/08 08:40:42 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:40:42 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for agent to check in and apply initial plan
2023/02/08 08:40:42 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:40:42 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for agent to check in and apply initial plan
2023/02/08 08:40:43 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for agent to check in and apply initial plan
2023/02/08 08:40:45 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:40:46 [INFO] [planner] rkecluster fleet-default/test-rke2: waiting: configuring bootstrap node(s) test-rke2-pool1-6d4f6844f5-q6x5v: waiting for agent to check in and apply initial plan
2023/02/08 08:40:55 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing
2023/02/08 08:41:25 [ERROR] error syncing '_all_': handler user-controllers-controller: failed to start user controllers for cluster c-m-fxfl28pp: ClusterUnavailable 503: cluster not found, requeuing

Is there any update on this issue or was someone able to resolve it?

[sebastianklein96]

Realizing I'm poking an old issue here, but do you remember what version of Rancher you used, @klauserber ? Still having the same issue using a local Rancher 2.7 instance provisioned with Hetzner Cloud credentials and with all the known fixes applied.

Also, if you used Rancher 2.6+ as well, where do you set the instance type in the new UI?

Try to enable the userDataFromFile flag in the settings by the pool. Otherwise the cloud-init file by rancher will not be copied/executed on the node.

[kaije5]

Also poking this issue. Have the same issue waiting for agent. Tried a lot of combinations RKE2/K3S different networks etc.
My rancher currently runs on a single node K3S install.

[ammediagmbh]

Try to pass https://github.com/hetznercloud/hcloud-cloud-controller-manager/releases/download/v1.20.0/ccm-networks.yaml in the Add On Config -> Additional Manifest.

I added the api key and network hardcoded in the yaml.

And use CLoud Provider: External and --cloud-provider=external as Additional Kubelet Args, Additional Controller Manager Args and Additional API Server Args.

Also add the Private IPS in TLS Names.

[mxschmitt]

#146

[nils-bq]

As Rancher has completely reworked the way that node driver UI extensions work, a simple adjustment of this existing codebase is unfortunately not sufficient to support RKE2. Therefore, I have written a new UI extension using the Rancher v3 extension API: https://github.com/bluquist/rancher-node-driver-hetzner

Maybe that is helpful for some others as well that are looking for a way to continue managing clusters on Hetzner cloud via Rancher with RKE2.