ProviderID tolerations to support Sidero, RKE, etc #140
Comments
|
Can you give me example please, I did not get your idea. ProviderID magic string uses by many controllers to find the node in the cloud. Do you need to label the nodes by special logic/rules? |
|
Hi there, as I am also affected by this (RKE2), let me try to explain what probably is @iAdanos problem, mine is it for sure: proxmox-cloud-controller skips setting node labels when a "foreign providerID" is set. Unfortunately, RKE2 sets the ProviderID itself, and therefore when trying to use proxmox-cloud-controller, we don't get it set the labels we need on the nodes in order to use proxmox-csi-plugin. Now that I have looked at it a bit more in detail, it seems that rke2 doesn't allow to change the providerID, so the approach of this project would not work at all. Just skipping setting the providerID is also not a solution as apparently proxmox-csi-plugin searches for the VM ID (Proxmox) in there to identify where in the cluster the VM is. I would like to do some tests disabling the default RKE2 Cloud-Controller-Manager, but I am not sure whether this project will act as a full replacement and disable the taints a host has if "no cloud controller manager" is running. Will let you know what the result is. Other than that, would it be viable to switch and to store the Proxmox node id into a label instead, and use this label also in proxmox-csi-plugin, maybe? This way, we could leave RKE2s own magic as it is and still maintain the important functionality. Thanks a lot in advance Christian |
|
I did some tests in my RKE2 cluster: I have set in the cluster configuration (in Rancher): This causes the nodes that are already existing to just stop the RKE-own cloud-controller-managers, but keep their With this configuration, I have tried to add a new node to the cluster. Following happens: No RKE2 cloud-controller-manager -> no providerID gets set by it. Proxmox-cloud-controller tries to do its thing, but: So, proxmox-cloud-controller-manager still refuses to set stuff for this node, now because "cloud-provider" name is (apparently automatically) set to "external". Therefore, no Then I tried the following: This unfortunately led an existing node, the bootstrap node to get unavailable for Rancher, as apparently there are only a few of "cloud-provider-name" settings allowed by RKE2. So setting it like this is not an option. I have found an example with the "out-of-tree AWS" Cloud-Controller-Manager here with a possibility to not set the rke2 option |
|
Hi, yep ProviderID is magic string, it can be set only once on join process. It is immutable value. I've never used the RKE but base on documentation, you need:
for controlplane: spec:
rkeConfig:
machineSelectorConfig:
- config:
disable-cloud-controller: true
kube-apiserver-arg:
- cloud-provider=external
kube-controller-manager-arg:
- cloud-provider=external
kubelet-arg:
- cloud-provider=externalworkers: spec:
rkeConfig:
machineSelectorConfig:
- config:
disable-cloud-controller: true
kubelet-arg:
- cloud-provider=external
|
|
I am curious, who sets the name as |
Using elementalOS as the base OS for the K8S nodes. You create a registration endpoint in Rancher, put a Cloud-Config behind, let Rancher create an iso image for you and use this to install VMs (automated, manual, PXE, Pulumi, whatever). With this, you create an "inventory of machines" of which Rancher will then just "grab some" depending on the label settings you supply when spinning up a new cluster. For these "machines", rancher creates random names. In my case, I have VMs called:
where "s" stands for "small" and "m" for "medium". These in the "inventory of machines" appear as follows:
...and these are being used as node names for the RKE2 cluster as well. So, the need of having "the VM name equal to the Kubernetes node name" may be another problem for using this CCM with Rancher, as this is definitely only the case if you build your cluster "manually" which (hopefully) nobody would do.... |
|
Thanks for the super quick reply. I quite do not understand some things yet, though...
What exactly does proxmox-csi-plugin expect to find in the providerID field? In the code I have found something about
Yeah, that's what I referred to in my last post yesterday. BUT, you are suggesting did I misinterpret this message? I thought, the CCM would skip this node because having
As mentioned in my other answer from a few minutes ago, this is a problem at least when using any of the Rancher "Cloud providers" which should be default nowadays when managing clusters dynamically. We cannot guarantee that the Kubernetes node name is equal to the VM name. Any chance to overcome this limitation? I mean, actually if we have the vmid, whouldn't this be enough to recognize the VM? |
|
Thanks, now i see that happens. The value The CCM must define the virtual machine So, we need to somehow pass the vmID to the VM. Do you know how Rancher can help with this? |
|
Labels. https://docs.rke2.io/advanced#node-labels-and-taints As far as I understand RKE2 sets labels on node cluster join and after that supposes, that labels should be managed by cluster admin, so if Proxmox CCM/CSI will set any other label not managed by Rancher, it will just keep it as is. |
That's understandable, as the CCM will have to attach/unattach and even move PVs from one VM to the other, it needs to know its VMID. Got it, thanks.
So, do I understand you correctly that the default procedure of the CCM is as follows:
...and as VMID does not change during VM lifetime, awesome! If that's right, I kind of have found a solution for that part: Elemental installer does set so called "MachineName" for each of the VMs that register themselves to be available to deploy a K8S on, it uses some UUID for the machine names, so that those machine names, from which the K8S node names will be derived (it's the same name in fact) have nothing to do with whatever you set as VM names in Proxmox. BUT: you can use SMBIOS data of the VM to set "Machine" labels (which will be K8S node labels later on, also) and even the "Machine Name". That's what I did: I have set machine name and some labels to reflect the SMBIOS "Product" field, and at VM creation I have added the VM name once in the VM name field and once in the "SMBIOS Options" -> "Product" field, in Proxmox. With this I end up with a Kubernetes cluster having nodes with equal names as the nodes' VM names in Proxmox, and also some good labels: As the VMs will in future for sure not be added by hand, as I am doing right now but employing any kind of automation (Ansible, Terraform, Pulumi), depending on the automation stack and its capabilities, I believe it would still be VERY important to have an alternative way of getting to know which VMID the particular VM is inside Proxmox. I would strongly opt for CCMs options:
The second, if applicable, seems to be the more reliable method, because as far as I have experienced by accident: it is possible to set several VMs with the same VM name in Proxmox, which for sure would cause a horrible behavior. Regarding |
|
I have not yet been able to get it to work, even with the "new" cluster where Node names are equal to VM names in Proxmox. Actually, I strongly believe the settings in The fact that my last test actually made the CCM try to do something let me to try it this way again. So, I have set the following: This will disable RKE2's own Cloud Controller Manager (the one that sets the providerID with rke://hostname). The documentation of RKE2 doesn't say much about the With this settings, I have killed one of my three nodes and let Rancher deploy a new one on a machine from the machine inventory. This node gets integrated in the K8S cluster, doesn't have a But unfortunately the Proxmox CCM does not really want to set one either: Looks like it erroneously interprets the non-existing providerID as foreign set one? Is that a bug in the CCM? |
|
Thank you @fibbs !
Yep, vmID change is very hard. And it possible only manually in proxmox shell. So, it is not our case.
Yep, you are right.
The kubelet can label the node and set the ProviderID during the join process by So we can pass the vmID through:
In my case: I am using terraform and it creates/updates But, after your last comment, you gave me an idea: I have a plan to make a release soon. It would be great to implement this feature before then. |
|
Your idea sounds great! Have you seen, we have written a message almost at the same time, so I suppose you haven't seen my newer message, such as I did not see yours. Do you have any idea why in my setup the CCM, now that I have successfully started it with a node not having a really scratching my head here |
|
instances.InstanceExists instances.InstanceShutdown methods call then node is unhealthy or shutdown, so as providerID is empty, the CCM cannot define the status of the VM on proxmox side (omitting unmanaged node). Try to remove the node resource and reboot the worker node. It seems to me the node is not properly initialized. PS. The error message not so obvious, i've fixed it already in my branch. |
|
I wanted to let you know, I finally got the CCM running in a Rancher managed RKE2 cluster. The process was:
The node will be in "Ready" state but Rancher will not let other nodes join the cluster, because the node will have the Problem was that I had the fully qualified domain name of the proxmox cluster in the settings. I am using an existing secret for the CCM to access the proxmox cluster: The CCM was not able to access the Proxmox cluster and timed out after some 5 minutes or so. Reason: CoreDNS is not running yet inside the cluster while the node still has the I now have changed the URL to contain the IP address, and suddenly it works as you described above:
Nevertheless, especially in combination with RKE2 etc, I would love to see a "non-CCM" mode of this software. Actually, the fact to set some labels (yes, providerID would have to stay untouched in such a mode, and another node label would have to be used for storing the VMID), the fact that this software is a CCM is actually a bit counter-productive: it needs network communication (when no DNS is working at the stage of no cloud-provider being active), it needs a secret (which should be deployed with a GitOps tool like Flux which is not yet working at this stage)... Even if the "no DNS" issue is possible to workaround, I would have to set the password / token for a user that has quite high permissions inside the Proxmox API as clear text in my cluster automation stack, in the cluster definition yaml. What is your opinion on that? |
|
Thank you for feedback.
Am I understanding correctly that you want to automatically label the node and removed kubernetes node resource when the nodes are scaled down or deleted on the Proxmox side, while keeping the providerID unchanged (non-Proxmox style)? If yes: CCM is designed to work with a single type of cluster and manage all the nodes within that cluster. If any node does not belong to the cloud, the CCM must also remove it from the Kubernetes cluster. I believe in hybrid cluster setups, so I implemented a CCM that supports multiple CCMs within a single cluster. It handles only the nodes that belong to Proxmox VE and skips others by checking the providerID for a specific magic string. We have already similar ideas kubernetes/cloud-provider#63, kubernetes/kubernetes#124885 like this. And we haven't started implementing them yet.. |
Feature Request
Add an option to tolerate some ProviderIDs on nodes, to permit an app labeling such nodes.
Description
To resolve the same issue as #111 for RKE2 and other K8S engines, it would be great to enable some flag or option to permit Proxmox-CCM application labeling nodes having ProviderID in labels.
For example, RKE labels nodes on bootstrap, but after that it assumes that cluster administrator will labels nodes on it's own.
An option to tolerate some provider IDs to not to skip such nodes would be great for such cases.
The text was updated successfully, but these errors were encountered: