High IO load on host with 3 KVM k3s master nodes

[cmonty14]

Hello,

I have made this observation on my host with k3s setup running with 3 master nodes; any master node is in a KVM running Alpine Linux.
There's a constant IO load on the disk caused by the relevant k3s-server-<n>; the IO load is low, but it's noticable.
This is the output of iotop:

Total DISK READ:         0.00 B/s | Total DISK WRITE:       361.01 K/s
Current DISK READ:       0.00 B/s | Current DISK WRITE:     403.49 K/s
    TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                                                                                                                                                              
4166041 be/4 root        0.00 B/s    0.00 B/s  0.00 % 27.95 % kvm -id 202 -name k3s-server-2 -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/202.qmp,server,nowait -mon chardev=qmp,mode=co~bridgedown,vhost=on -device virtio-net-pci,mac=7A:A4:D6:4F:D3:0D,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102 -machine type=pc+pve0
 189489 be/4 root        0.00 B/s    0.00 B/s  0.00 % 20.10 % kvm -id 203 -name k3s-server-3 -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/203.qmp,server,nowait -mon chardev=qmp,mode=co~bridgedown,vhost=on -device virtio-net-pci,mac=BA:F4:94:95:4F:31,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102 -machine type=pc+pve0
4169876 be/4 root        0.00 B/s    0.00 B/s  0.00 %  4.05 % kvm -id 201 -name k3s-server-1 -no-shutdown -chardev socket,id=qmp,path=/var/run/qemu-server/201.qmp,server,nowait -mon chardev=qmp,mode=co~bridgedown,vhost=on -device virtio-net-pci,mac=FA:D1:FE:FB:51:CB,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=102 -machine type=pc+pve0
 458108 be/3 root        0.00 B/s    3.54 K/s  0.00 %  1.34 % [jbd2/loop1-8]
    485 be/3 root        0.00 B/s    0.00 B/s  0.00 %  1.32 % [jbd2/sda1-8]

Question:
Is it normal for a k3s cluster setup that the processes are causing IO load although there's no "productive" pod?

 kubectl get pods -A
NAMESPACE     NAME                                      READY   STATUS      RESTARTS   AGE
kube-system   coredns-7448499f4d-s26zm                  1/1     Running     3          3d23h
kube-system   helm-install-traefik-crd-5vcj2            0/1     Completed   0          3d23h
kube-system   helm-install-traefik-s85mr                0/1     Completed   1          3d23h
kube-system   local-path-provisioner-5ff76fc89d-lwltx   1/1     Running     3          3d23h
kube-system   metrics-server-86cbb8457f-hgmwn           1/1     Running     3          3d23h
kube-system   svclb-traefik-2v4xl                       0/2     Evicted     0          11m
kube-system   svclb-traefik-8qm7j                       0/2     Evicted     0          20m
kube-system   svclb-traefik-fsql7                       2/2     Running     0          4h51m
kube-system   svclb-traefik-r7gxg                       2/2     Running     2          3d22h
kube-system   svclb-traefik-s64p8                       2/2     Running     0          4h34m
kube-system   traefik-97b44b794-2ns8q                   1/1     Running     3          3d23h
portainer     portainer-59f798c579-hw8hd                1/1     Running     0          38h

THX

[brandond]

You didn't include any info on what you're using for your datastore, but since you've got a 3-node cluster I'm guessing you're using managed etcd? Etcd is far more demanding of your storage than sqlite or external sql, as it triggers a full fsync to disk every time there is a write to the datastore. Since Kubernetes is constantly updating internal status for things like keepalives, health checks, component leases, etc. you will see a fairly high base IO load even with nothing going on in the cluster.

For more information, see:
https://rancher.com/docs/k3s/latest/en/installation/installation-requirements/resource-profiling/#:~:text=IOPS%20and%20memory%20requirements

[cmonty14]

Your assumption is 100% correct: I'm using managed etcd.
As said, all nodes of the cluster are running in KVM, and the cluster currectly consists of

• 3 server nodes
• 2 worker nodes

Now, I can consider a different setup, means High Availability with an External DB.
Would this result in IO load channeled to the external DB in the same "size" compared to Embedded DB?

Or would External DB reduce the IO load?

[brandond]

In general, IO for SQL backends is lower than etcd. Every write to etcd needs to be written to disk on each node, so if you have three etcd servers then that's three nodes writing a copy of every change. If you have a single database server, then the write only needs to be performed once. Database servers are also usually a bit less eager to write things to disk, and won't force a full fsync for every insert.