Improvement Ideas for Longhorn disks on Cloud Volumes

[andi0b]

This project has support for Longhorn and also mounting a Hetzner Cloud Volume to /var/lib/longhorn as it's default disk. I think this feature is not really great how it is right now (see #1194 and #1195) , I wanted to share some ideas I've collected over the past few days.

Maybe I'm going to submit a PR for those changes if I manage to do it.

Why use Longhorn on Cloud Volumes in the first place?

Possible providers for Persistent Volumes (PVs) are:

• Cloud Volume CSI, which bring some limitations: No snapshots, no consistent backups, at least 10 GB in size, no ReadWriteMany, in the past there were reliability issues when moving pods with volumes to other nodes
• Longhorn: supports Snapshots, ReadWriteMany, backups, replication, ...
• k3s local storage: no replication, also no snapshots, backups, ...

Longhorn has quite some benefits over the other solutions. It's easily possible to snapshot/backup volumes, move them between nodes and pods without longer interruptions.

Also a hybrid scenario is possible, to move/sync longhorn volumes to other cloud providers and even on premise. Although performance implications are expected if the network is not fast enough (non fiber).

What's missing now

The possibility to add more than one disk to Longhorn, for example I would like to do:

• Provision one Longhorn disk with the label "ssd" on the VMs root filesystem, because it's much faster than Cloud Volumes
• Provision another (or more than one) Longhorn disk on a Cloud Volume, because it's the only option for multi-TB volumes
• Shrinking of Lonhorn disks can be achieved by adding another smaller Cloud Volume, then drain the larger disk via Longhorn Admin UI (Longhorn will move the replicas to the new disk), then delete the bigger Cloud Volume

What I think should be added to this project

Extend node pools with volume support:

• Being able to add Cloud Volumes to Agent and CP nodes and mount them to a specified path
• Support growing, adding and removing volumes
• Think about a solution for delete-protection of those Cloud Volumes (deleting those volumes could be a manual process)
• and for a solution to re-use existing volumes for new nodes in the case someone is moving to a new node pool but wants to attach existing Cloud Volumes to new nodes. (Longhorn will automatically pick up non-empty disks and all the replicas on them).

Maybe even auto-configuration for multi-disk Longhorn

• Allow setting the annotation node.longhorn.io/default-disks-config to nodes in the pool, to make longhorn pick up the volumes

Alternative ideas:

• Expose the node pools via terraform out vars (Outputs for Kubernetes Terraform provider #1165), especially the server id of the created nodes, so someone can add own volume logic to the terraform outside the terraform-hcloud-kube-hetzner module

[aleksasiriski]

Thanks for writing this out, here's my take on this:

1. Hetzner Volumes are great for smaller clusters if you need HA volumes (it's CEPH replicated storage), also it's MUCH cheaper even though it's slower than nodes storage (I use it as a temp volume of 100gb for downloading some random files that get moved elsewhere).
2. Local storage is extremely fast and doesn't impact performance in any way since it's just Kubernetes mounting to a path and locking the pod to the node.
3. Longhorn is very heavy on the system, it practically mandates that you have nodes with at least 4gb of ram and run almost nothing on them. This is especially true if the volumes are write heavy, for example database, since it's writing to multiple locations at the same time and over the network.

All of those have their usecases, all should be supported. Now...

1. Hetzner Volumes are a little buggy sometimes but work without effort from our side, it's all Hetzner CSI operator
2. Local PV is also pretty easy to setup, since it's integrated with K3s and we just have to flip the flag. It does have it's limitations (no quota limits for storage and no moving of data)
3. Longhorn is VERY complex and as far as I've played with it, it's not very declarative. It assumes you'll be using the GUI and creating custom CRDs for it to configure it to suit your needs. This is very incompatible with the nature of Terraform and this project and most of the features are out of scope. That is exactly why most people opt out of using our provided Longhorn option and just end up deploying their own using Helm.

IMO Longhorn configuration shouldn't be half of this projects code, it wasn't designed to be configured via Terraform and I don't think we should force it to be that way. Most of the features you described would, at least for me, fall into a category of a "Longhorn Kubernetes Operator" or to be integrated into Longhorn itself. As another solution, it might be more practical and easier to use something like Ansible to achieve the manual work you were describing.

In conclusion, it will be fine to implement outputs for nodes so that other modules can use them as well as better labeling system for the nodes, but the complex logic around Longhorn should be kept separate from this repository. Terraform wasn't made to achieve half automatic half manual work, it's supposed to do infrastructure as code declaratively and predictably saving everything to a state file.

Since I'm just a contributor to this project, don't take my word as something that is 100% right. I would gladly have @kube-hetzner/core comment on this as well.

[mysticaltech]

@aleksasiriski Fully agree with you and back your judgement. Apart from node level configs, that really do make 70% of use cases much easier, I think we do not need to focus more on longhorn and should fix the docs to advice using their own helm deployment and node-level setups for advanced use cases, as this project is definitely not meant for that kind of customizations.

@andi0b Do checkout one of our core member @ifeulner's guide on using Longhorn with this project, it's quite good IMHO. https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b

[andi0b]

Do checkout one of our core member @ifeulner's guide on using Longhorn with this project, it's quite good IMHO. https://gist.github.com/ifeulner/d311b2868f6c00e649f33a72166c2e5b

@mysticaltech Thanks for the link. But I think it's just barely touching the surface.

One additional thing this post does mention: Setting the node.longhorn.io/default-disks-config node annotation. Node pools currently allow setting labels, but not annotations. Another thing that would be nice. Just saying, and I know I can make a PR ;).

[mysticaltech]

@andi0b That's what I was going to say, you beat me to it, the best way to convey your vision for improvements is through a PR :)

[ifeulner]

Storage management is quite an exhaustive and individual requirements-oriented topic... so it's quite challenging if not even really useful to cover this in a more generic-oriented project like this one.
Storage management needs maintenance, and I think maintenance is out-of-scope for any terraform-based project anyway... but longhorn can be pretty well configured/maintained via k8s resources... and can be automatized then by using flux or ArgoCD.

Hetzner volumes are the only way to get more / significant storage to the cluster, but the CSI driver is (maybe was, haven't used it a long time anymore) unstable. Therefore this project mounts the Hetzner volumes directly to the nodes (no CSI driver needed, therefore).

Currently, this is limited to one volume per node, but it could be extended easily (but there is a Hetzner limit of a maximum of five mounted volumes per node afaik).

Then you can design your longhorn volumes on top of these (mirroring, distribution, etc.). This is very requirements-specific, therefore adding the configuration via annotations, CRDs, etc afterward makes quite a sense...

Just my 5cts.. ;)

[andi0b]

Then you can design your longhorn volumes on top of these (mirroring, distribution, etc.). This is very requirements-specific, therefore adding the configuration via annotations, CRDs, etc afterward makes quite a sense...

Exactly. After the cluster, nodes and networking is created it's just like any other Kubernetes cluster. And the whole Terraform ecosystem can be used to customize it.

I think even some currently built in features could be removed and just replaced by documentation and code samples (Longhorn, Nginx, ...)

[valkenburg-prevue-ch]

The last time I tried, I did not find a pure tf solution for annotating the nodes with through k3s configuration. I currently label them, and then after kube-hetzner completed, I run a local bash kubectl command to annotate certain nodes of they have a certain label. Pretty disappointing. I would cheer if you find a better way!

…

-------- Oorspronkelijk bericht -------- Op 22 feb. 2024 16:51, schreef K. N. :

***@***.***(https://github.com/andi0b) That's what I was going to say, you beat me to it, the best way to convey your vision for improvements is through a PR :) — Reply to this email directly, [view it on GitHub](#1196 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/A3W7X2MLKHFPKUV42AI7BYLYU5SOTAVCNFSM6AAAAABCOILDXSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKNJYGYZDG). You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[mysticaltech]

@valkenburg-prevue-ch We do run already quite a few local-exec, so that would work to execute your annotation script. @andi0b FYI on this approach.

We could also use the kubernetes provider, but it's more lines of code. However, it's an interesting technique to keep in mind. More info here: https://chat.openai.com/share/289ed74f-1677-4ed3-993a-63710eb8f916

[andi0b]

I didn't realize it's that complicated. Should be out of scope again then. Another thing that can be solved with outvars and custom terraform around this project.

[valkenburg-prevue-ch]

Here's the gist of it:

data "kubernetes_nodes" "all" {}

resource "null_resource" "longhorn_node_tags" {
  depends_on = [data.kubernetes_nodes.all]
  triggers = {
    "node_labels" : join(",", [for node in data.kubernetes_nodes.all.nodes : join(",", [for key, value in node.metadata.0.labels : "${key}=${value}"])])
  }
  provisioner "local-exec" {
    command     = file("${path.module}/files/longhorn/tag-nodes.sh")
    interpreter = ["/bin/sh", "-c"]
    environment = {
      KUBECONFIG = var.kubeconfig_path
    }

  }
}

and then tag-nodes.sh:

# only node
kubectl annotate --overwrite nodes --selector "kube-hetzner/use-node-storage=true" "node.longhorn.io/default-node-tags=[\"node-storage\"]"

# only large
kubectl annotate --overwrite nodes --selector "kube-hetzner/has-ssd-storage=true" "node.longhorn.io/default-node-tags=[\"large-storage\"]"

Surely this can rather trivially be generalised, by making a label like something as kube-hetzner-label-to-annotations and its value some parsable (json?) string which could can be an arbitrary set of annotations. It is a bit abusive, but k3s and actually k8s in generally do not support annotating nodes at creation: k3s-io/k3s#2626

There is also https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/resources/annotations , but I'm not sure it supports only adding annotations without removing annotations absent from its declaration.

[valkenburg-prevue-ch]

That's a good point. It's really time for a full blown docusaurus website for this project. But that takes a lot of added maintenance, nothing is worse than faulty outdated documentation.

…

-------- Oorspronkelijk bericht -------- Op 22 feb. 2024 18:11, schreef andi0b :

> Then you can design your longhorn volumes on top of these (mirroring, distribution, etc.). This is very requirements-specific, therefore adding the configuration via annotations, CRDs, etc afterward makes quite a sense... Exactly. After the cluster, nodes and networking is created it's just like any other Kubernetes cluster. And the whole Terraform ecosystem can be used to customize it. I think even some currently built in features could be removed and just replaced by documentation and code samples (Longhorn, Nginx, ...) — Reply to this email directly, [view it on GitHub](#1196 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/A3W7X2NY4UIYS6KURAC7I53YU535PAVCNFSM6AAAAABCOILDXSVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKNJZGU4DG). You are receiving this because you were mentioned.Message ID: ***@***.***>

[aleksasiriski]

We should do this with v3 release

[mysticaltech]

@valkenburg-prevue-ch @aleksasiriski Definitely for v3, and we have so much "support" data now that it's going to be really good for a custom gpt and github bot that will be able to reply to 90% of issues. I have this in the pipe.