Parsing error while loading KubeletConfiguration object from configuration file

[ehlesp]

Environmental Info:
K3s Version:
k3s version v1.22.3+k3s1 (61a2aab)
go version go1.16.8

Node(s) CPU architecture, OS, and Version:
Linux k3sserver01 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux

Cluster Configuration:
One server node

Describe the bug:
Parsing issue with kubelet configuration file. The k3s process is unable to read KubeletConfiguration objects, returning an error when it finds a ',' character in the file.

Steps To Reproduce:
Prepare a kubelet.config file like the one below in an accesible path for the k3s service like /etc/rancher/k3s.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 30s,
shutdownGracePeriodCriticalPods: 10s

Then, either add the file as a command line argument (--kubelet-arg="config=/etc/rancher/k3s/kubelet.config") or as a parameter in a customized /etc/rancher/k3s/config.yaml file (kubelet-arg: "config=/etc/rancher/k3s/kubelet.config").

In my case, I used a customized config.yaml file as the following one. Notice the kubelet-arg specified in it.

# k3sserver01

cluster-domain: "deimos.cluster.io"
tls-san:
    - "k3sserver01.deimos.cloud"
flannel-backend: host-gw
flannel-iface: "ens19"
bind-address: "0.0.0.0"
https-listen-port: 6443
advertise-address: "10.0.0.1"
advertise-port: 6443
node-ip: "10.0.0.1"
node-external-ip: "200.0.0.1"
node-taint:
    - "k3s-controlplane=true:NoExecute"
log: "/var/log/k3s.log"
kubelet-arg: "config=/etc/rancher/k3s/kubelet.config"
disable:
    - metrics-server
    - servicelb
protect-kernel-defaults: true
secrets-encryption: true
agent-token: "someLongPassword"

The last step was installing the K3s service as a server node.

wget -qO - https://get.k3s.io | INSTALL_K3S_VERSION="v1.22.3+k3s1" sh -s - server

• Installed K3s: v1.22.3+k3s1

Expected behavior:
The installation finishes and the K3s service starts correctly.

Actual behavior:
The installation finishes but the K3s service is unable to run properly due to an error while trying to load the kubelet.config file.

Additional context / logs:
The K3s service prints the following error log line about every 10 seconds.

E1128 12:21:51.382209    4795 server.go:205] "Failed to load kubelet config file" err="failed to load Kubelet config file /etc/rancher/k3s/kubelet.config, error failed to decode: v1beta1.KubeletConfiguration.ShutdownGracePeriodCriticalPods: ShutdownGracePeriod: unmarshalerDecoder: time: unknown unit \"s,\" in duration \"30s,\", error found in #10 byte of ...|od\":\"30s,\",\"shutdown|..., bigger context ...|KubeletConfiguration\",\"shutdownGracePeriod\":\"30s,\",\"shutdownGracePeriodCriticalPods\":\"10s\"}|..."path="/etc/rancher/k3s/kubelet.config"

The K3s service loads the file and runs as expected only after removing the ',' from kubelet.config.

Backporting

• Needs backporting to older releases

[brandond]

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
shutdownGracePeriod: 30s,
shutdownGracePeriodCriticalPods: 10s

This isn't a K3s issue - the yaml you provided contains in invalid value, and Kubernetes refuses to use it. 30s, is not a valid duration string; durations do not include commas.

See https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ in particular:

shutdownGracePeriod meta/v1.Duration shutdownGracePeriod specifies the total duration that the node should delay the shutdown and total grace period for pod termination during a node shutdown. Default: "0s"

[ehlesp]

This isn't a K3s issue - the yaml you provided contains in invalid value, and Kubernetes refuses to use it. 30s, is not a valid duration string; durations do not include commas.

See https://kubernetes.io/docs/reference/config-api/kubelet-config.v1beta1/ in particular:

shutdownGracePeriod meta/v1.Duration shutdownGracePeriod specifies the total duration that the node should delay the shutdown and total grace period for pod termination during a node shutdown. Default: "0s"

Sorry to insist but I just want to understand where this issue comes from. Below I copy an example taken from the official Kubernetes documentation about setting Kubelet parameters via a config file.

apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
address: "192.168.0.8",
port: 20250,
serializeImagePulls: false,
evictionHard:
    memory.available:  "200Mi"

As you can see in the yaml above, there are several commas acting as separators, not as part of the value on each parameter. So, given that kubelet-arg is the argument through which I can reference a config file with Kubelet parameters to K3s, shouldn't the format of this particular file comply with what's shown in the official documentation?

[brandond]

The upstream docs are wrong, that's not a valid configuration file either. Commas only act as separators within a bracketed list: [1, 2, 3]. Syntactically it's even less valid than yours, yours is at least valid YAML. If you wanted to open a PR to the Kubernetes docs, they'd probably appreciate it.

https://go.dev/play/p/Gb0VEVODrxq

[ehlesp]

The upstream docs are wrong, that's not a valid configuration file either. Commas only act as separators within a bracketed list: [1, 2, 3]. Syntactically it's even less valid than yours, yours is at least valid YAML. If you wanted to open a PR to the Kubernetes docs, they'd probably appreciate it.

https://go.dev/play/p/Gb0VEVODrxq

Many thanks for your clarification, no wonder I found such format contradictory the first time I saw it. I'll consider the PR, when I have the time.

[simonspg]

I am writing a big Thank You for this lucid description. Apparently, k3s is migrating to configuration files, which is fine; but it would be nice if the documentation kept up. But documentation is hard to keep up to date. Thank you again.

[brandond]

We are not migrating to config files, but they are supported.

[simonspg]

I said that because trying to add --shutdownGracePeriod as a command line argument results in "Failed to parse kubelet flag" err="unknown flag: --shutdownGracePeriod", whereas the config file is not rejected. I am in the process of testing to see if it actually changes behavior.

[brandond]

That's because the kubelet doesn't have flags for everything that's possible via the kubelet config file.

That does not mean that "k3s is migrating to configuration files" though, this is just how the Kubernetes kubelet works, regardless of what distro you're using.

[simonspg]

Got it. k8s in general, not k3s in particular. And, no surprize, it works as expected. And someday node fencing.

[bchill]

FWIW, I prefer config files. 😄