Support extra configuration for K3S installer

[Shakahs]

The K3S installer has configuration options that are not directly exposed via hetzner-k3s. It would be very helpful if users could pass arbitrary environment variables or command line strings to the K3S installer to utilize those configuration options.

hetzner-k3s already allows users to provide additional configuration parameters for Kubernetes components (kubelet_args, kube_proxy_args, etc.) so this would just be continuing that pattern.

[vitobotta]

Hi, can you give me some example of extra args that you find useful?

[Shakahs]

External Datastore Configuration Parameters

--datastore-endpoint
--datastore-cafile
--datastore-certfile
--datastore-keyfile

[Dani4kor]

i am interesting too in ability to deploy and configure external database in hetzner_k3s while creating cluster, it will be really helpfull

[vitobotta]

Thanks @Shakahs for the clarification. Added to my todo list.

Out of curiosity, how are you managing the external datastore?

[Shakahs]

@vitobotta I utilize managed Postgres from Crunchy Data.
This enables me to run a stateless cluster with all persistent data on managed services (Crunchy for datastore and Hetzner for persistent volumes).
I also consider managed Postgres to be more reliable than etcd in a Placement Group.
My cluster is in Ashburn so latency to Crunchy in AWS us-east-1 is not a factor.

[vitobotta]

@Shakahs thanks for the clarification. Sounds like an interesting solution! Once I have made this change it would be nice if you could write a quick tutorial or something about it.

[Shakahs]

@vitobotta Yes I would be glad to do so.

[vitobotta]

@Shakahs I am looking to test this. Which Crunchy Data plan do you use and how large is the cluster using it? Can you give me some details? Thanks!

[Shakahs]

I'm using the hobby-0 plan with 10GB of storage and HA turned off. My cluster is small: 1 master, 3 workers. From what I've seen K3s/Kine doesn't put much load on Postgres, the dataset is tiny (5.6 megabytes for my cluster) and the query rate is low by Postgres standards. Much of it is just K8s establishing locks on resources. I would expect Postgres resource usage to stay low even on bigger clusters.

…

On Mon, Nov 20, 2023 at 9:15 AM Vito Botta ***@***.***> wrote: @Shakahs <https://github.com/Shakahs> I am looking to test this. Which Crunchy Data plan do you use and how large is the cluster using it? Can you give me some details? Thanks! — Reply to this email directly, view it on GitHub <#229 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAC7JHWWMYXFPCNP4JF4NJLYFOF27AVCNFSM6AAAAAAZFIDYTOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJZGQ4DSMBZHE> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[vitobotta]

Thanks. In the meantime I found https://neon.tech/ which sounds better because it's serverless, you pay for what you consume, it scales automatically and it's highly available by default :)

[Shakahs]

I did test Neon and it does work with K3s. Neon is the better choice for testing environments because they provision instantly and Crunchy takes 10+ minutes, but Neon is more expensive than Crunchy for a production deployment.

K3s/Kine maintain an open connection to Postgres at all times from each master, so Neon keeps the database active at all times instead of scaling it to 0.
Because K3s/Kine do not benefit from Neon's serverless capability, and because Neon charges more for Postgres resources than Crunchy, I opted to use Crunchy.
For reference, the Hobby-0 plan is $10/month at Crunchy (with 2 vCPU, 512MB of RAM, 10GB disk). For 24x7 use, Neon would start at $20/month (with 0.25 vCPU, 1024MB of RAM, 10GB disk).

[vitobotta]

But Neon is HA by default, at least that's my understanding. I guess I'll try Crunchy too and see. What put me off with Crunchy is the warning that with the hobby plans there is a risk of low CPU performance due to the credit system. Have you had any problems at all with it? Also how many connections does the hobby-0 plan support?

[vitobotta]

@Shakahs Crunchy is pretty easy too and works fine and I agree that the specs are probably better for the same price. I will use this since you have had good experience with it :)

[vitobotta]

Another question, are you using these parameters? If yes, how?

--datastore-cafile
--datastore-certfile
--datastore-keyfile

[Shakahs]

I am not using the certificate parameters, this is the only change I made to hetzner-k3s:

-curl -sfL https://get.k3s.io | INSTALL_K3S_VERSION="{{ k3s_version }}" K3S_TOKEN="{{ k3s_token }}" INSTALL_K3S_EXEC="server \
+curl -sfL https://get.k3s.io | K3S_DATASTORE_ENDPOINT="postgres://k3s:xxxxxxxxxxx;[email protected]/k3s"  INSTALL_K3S_VERSION="{{ k3s_version }}" K3S_TOKEN="{{ k3s_token }}" INSTALL_K3S_EXEC="server \

I think the warning about CPU credits is good transparency by Crunchy. They appear to provision a separate EC2 instance for each Postgres instance, and the hobby plans align with the specifications of EC2 burstable instances which do in fact use a CPU credit system. I have used EC2 burstable instances with great success in the past, they are perfect for saving cost on workloads with low or intermittent resource usage.

Because K3s/Kine put such a small load on Postgres I do not think CPU credits are a problem and I have not had issue with them during my usage.

Unlike other providers, I do not believe Crunchy imposes an artificial connection limit. Your only limitation is whatever PGBouncer and Postgres itself can handle with the CPU and RAM allocation of the instance.

Neon's serverless architecture does make it HA be default, since any of their compute instances can service your request, but I'm not convinced that K3s even needs HA.

If the Postgres instance fails and Crunchy needs to replace it, which takes 10 minutes to provision, K8s workloads continue to function. The K8s API will not be available, and certain functions like autoscaling will not work, but whether this matters depends on the workload.

[vitobotta]

Good points 👍

[vitobotta]

Wow I broke CrunchyData :D I was doing some testing with many nodes and it crashed and I couldn't recover it. I can't restart or resize it and get 500s from the control panel. I am gonna try the same thing with NeonDB now.

[vitobotta]

Ref project card: https://github.com/users/vitobotta/projects/4/views/1?filterQuery=extra&pane=issue&itemId=59547211