kube-apiserver unhealthy because of etc failed (reason withheld)

[kenho811]

I installed rke2 on a debian bookworm VM.

I notice that every now and then, my kube-apiserver becomes unhealthy. I checked the kubernetes logs and notice this.

Message:         Liveness probe failed: Error from server (InternalError): an error on the server ("[+]ping ok\n[+]log ok\n[-]etcd failed: reason withheld\n[+]kms-providers ok\n[+]poststarthook/start-encryption-provider-config-automatic-reload ok\n[+]poststarthook/start-kube-apiserver-admission-initializer ok\n[+]poststarthook/generic-apiserver-start-informers ok\n[+]poststarthook/priority-and-fairness-config-consumer ok\n[+]poststarthook/priority-and-fairness-filter ok\n[+]poststarthook/storage-object-count-tracker-hook ok\n[+]poststarthook/start-apiextensions-informers ok\n[+]poststarthook/start-apiextensions-controllers ok\n[+]poststarthook/crd-informer-synced ok\n[+]poststarthook/start-service-ip-repair-controllers ok\n[+]poststarthook/rbac/bootstrap-roles ok\n[+]poststarthook/scheduling/bootstrap-system-priority-classes ok\n[+]poststarthook/priority-and-fairness-config-producer ok\n[+]poststarthook/start-system-namespaces-controller ok\n[+]poststarthook/bootstrap-controller ok\n[+]poststarthook/start-cluster-authentication-info-controller ok\n[+]poststarthook/start-kube-apiserver-identity-lease-controller ok\n[+]poststarthook/start-deprecated-kube-apiserver-identity-lease-garbage-collector ok\n[+]poststarthook/start-kube-apiserver-identity-lease-garbage-collector ok\n[+]poststarthook/start-legacy-token-tracking-controller ok\n[+]poststarthook/aggregator-reload-proxy-client-cert ok\n[+]poststarthook/start-kube-aggregator-informers ok\n[+]poststarthook/apiservice-registration-controller ok\n[+]poststarthook/apiservice-status-available-controller ok\n[+]poststarthook/kube-apiserver-autoregistration ok\n[+]autoregister-completion ok\n[+]poststarthook/apiservice-openapi-controller ok\n[+]poststarthook/apiservice-openapiv3-controller ok\n[+]poststarthook/apiservice-discovery-controller ok\nlivez check failed") has prevented the request from succeeding

Apparently it is related to etcd failure, but I am not too sure how to debug this.

Full log attached below.

kube-apiserver-rke2-master-prod.17d2eca167fb7aca.txt

=========
Observation

I observe that the warning event occurs mostly when new pods are being created.

I have an Airflow Scheduler which schedules new Kubernetes Pods every now and then.

Below is a screenshot of the events in my cluster.

When I suddenly schedule a lot of pods, the WARNING event occurs

[brandond]

Have you checked the etcd pod for restarts, or looked at the etcd pod logs? You might also consider looking at the apiserver pod logs?

[kenho811]

Hi @brandond

Thank you for your help.

With some investigation, I realise that I am running etcd on a slow HDD.

etcd timeout and apiserver cannot get a response and becomes unhealthy.

After I moved the etcd database to a dedicated NVME SSD, the timeout error is gone.

I observe the apiserver (along with the kube-scheduler etc. etc.) have stayed healthy after the fix.

[kenho811]

Moved etcd database to a dedicated NVME SSD and the issue is resolved.

[cengizhanR]

@kenho811 how did you move etcd database to a dedicated NVME SSD ? I am facing the same problem. Can u share the solution step by step?

[kenho811]

Hi @cengizhanR

I am running RKE2 inside a VM in Proxmox ( A hypervisor).

Via VM's setting in proxmox, I added an SSD storage.

Once I shell-ed into the VM, I mount the SSD storage to the location of the etcd database folder.

This is done via the below.

Your new storage will appear as a block device.

Use lsblk to list block.

Then format it with file system (e.g. ext4).

Then add it to /etc/fstab/.

(Follow the standard procedure online to format a new device, basically)

As for the location of the etcd data, If I remember correctly, it is in some /var/lib/rancher/server/db folder.

In short, the key is to mount your SSD to that path where rke2 stores the etcd data (the db folder).

[cengizhanR]

oh I understand, thank you very much @kenho811