Agent registration does not create node-password secret

[braunsonm]

Environmental Info:
RKE2 Version: 1.26.5+rke2r1

Describe the bug:
I noticed during a disaster recovery that RKE2 is no longer storing any node-password secrets on my cluster. It is still referenced in the documentation however, has this been removed? Or is there something strange going on with my cluster specifically?

Interestingly my agents aren't joining with the error Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry. But all my server nodes did fine and they still do not contain any node-password secrets in the cluster.

Steps To Reproduce:

• In my case I'm not sure what I did to cause this, I'm joining nodes together as per the quickstart and no node-password is created.

Expected behavior:
A node-password is created

Actual behavior:
kubectl get secrets -n kube-system | grep node-password

No entries listed on a 5 node cluster.

Additional context / logs:

[brandond]

The docs are correct. The error you're showing also indicates that the secrets exist.

Where specifically are you looking to see that the node password secrets do not exist?

What is the specific disaster-recovery process that you are testing here? Is it possible that you've got some inconsistent state due to how you're restoring things?

[braunsonm]

Hey @brandond We had to restore due to losing quorum. So I went through these steps here: https://docs.rke2.io/backup_restore?_highlight=reset#cluster-reset

The servers were able to rejoin after this but there are no node-passwords created and the agents are unable to join. Even after the agent VMs have been completely recreated (same hostname though). They do not show in get nodes and the command I listed above to list secrets have zero node-password secrets.

Interestingly enough, my other healthy cluster only has 2 node-password secrets for a total of 5 nodes and has been healthy for months/years.

Where specifically are you looking to see that the node password secrets do not exist?

kubectl get secrets -n kube-system | grep node-password

Anything you'd suggest me looking at? Perhaps even a way to generate the secret myself to at least get these agents to join?

[braunsonm]

I was able to find out why my agents weren't joining - because I was recovering from a disaster and had to restore a server node first, Rancher's webhook was not running (but was installed) because it does not tolerate the server taints.

Since Rancher's webhook was not running, it was actually blocking RKE2's ability to check/create the node-password secret. As soon as I tolerated the taints to get that pod running, the agent joined and a node-password was created. It does not explain why the servers were able to join and did not create node-passwords but it at least allows me to continue.

It's rather bad that a misbehaving ValidatingWebhook can interfere with kube-system and therefore affect RKE2's ability to join nodes to the cluster.

[brandond]

what release of Rancher are you on? I remember at some point in the past it set up the webhook in downstream clusters with a somewhat excessive number of resources that it would interfere with, but i think that was fixed at some point.

You're on a somewhat dated release of RKE2, so I suspect you might be on an old release of Rancher as well?

[braunsonm]

This is actually not a problem on the downstream clusters, it's a problem on our RKE2 cluster that is running the Rancher UI. We are running the latest version (2.8.1)

FYI do you know why/how the server nodes are able to join without a node-password?

[brandond]

Ah interesting. Yeah, in the case of the local cluster you might need to do additional work when restoring to disable validating webhooks until the pods behind them are up. This is a common issue with tools that enforce security via webhooks that run within the cluster, you can get into chicken-and-egg scenarios where the tools don't work because the tools aren't working.

Servers validate their own node passwords, as they have the keys to the kingdom and join before the apiserver is up to check the secret. The secret is checked later, and a warning raised if it doesn't match.