Nodes becoming unschedulable during k8s upgrade

[sgaide]

Hi guys, here is the situation:

• I use French OVH k8s managed service
• I install longhorn to use the nodes disk, these are the root disk, I know this is dangerous ;-)
• When I first install Longhorn, all node disks are recognized and used
• During a k8s upgrade, OVH system will re-image the nodes, one by one
• After a node is re-imaged and ready again the disk UUID changes, so that when longhorn restarts it will not recognize the disk, making the node "unschedulable"
• When the last node is upgraded and re-imaged all volumes are faulted, all nodes are unschedulable

I have read the documentation about upgrading when using AWS/GCP/Azure managed services, but solutions there do not seem to apply to OVH case (for example i can not create a new node pool with new version of k8s, i'm stuck with current version).

My question here is: is there an (automatic) way to make longhorn use the newly seen disk, as if it was "added" ?

What I have tried with success is to manually change the disk UUID (in /var/lib/longhorn/longhorn-disk.cfg) using the old one, after each node upgrade. Then replicas are rebuilt as necessary, and the upgrade proceed correctly. Of course this is not usable in "real life", but it proves that an upgrade could be possible with no service disruption.

Thanks a lot for any idea about this,

cheers

Sebastien

[STYNET]

Hello Sébastien,
I'm also using the OVH k8s managed service and I'm testing the Longhorn solution for using RWX volumes.
After the kubernetes update, I have the same problem as you.
It seems that nodes are considered stateless at OVH, so it may not be the best way to store data in /var/lib/longhorn of worker nodes.
Given that your post is about 1 year old, perhaps you'd have some interesting tips to share about your feedback?
Thanks in advance for your reply.
a+
Stéphan

[sgaide]

Hello Stéphan,

Unfortunately we have found no other solution than to progressively move away from longhorn and use hosting service native volume management.

cheers

S.