Customizing the Containerd state directory

[BayanAzima]

Is your feature request related to a problem? Please describe.
I would like to change containerd's state directory in its config.toml from "/run/containerd" to "/apps/rancher/containerd" since this mount has more space for ephemeral-volumes.
docs including state property: https://github.com/containerd/containerd/blob/master/docs/man/containerd-config.toml.5.md
this is where i think the value is set: https://github.com/k3s-io/k3s/blob/master/pkg/agent/config/config.go#L455

Describe the solution you'd like
Is it possible to configure k3s v1.19.7+k3s1 with a new state location. Also could anyone provide me with high level steps to make the change on an already existing cluster.

Describe alternatives you've considered
Nothing atm, I was curious on what you all recommend i do to change containerd's state directory.

Additional context
running k3s v1.19.7+k3s1

Thank you.

[brandond]

Customizing the containerd configuration is covered in the docs:
https://rancher.com/docs/k3s/latest/en/advanced/#configuring-containerd

[BayanAzima]

brought up a single node test cluster to observe the behavior of containerd when the state directory is changed via the tmpl file.

[root@sampleserver containerd]# ls
config.toml  config.toml.tmpl
[root@sampleserver containerd]# cat config.toml
# runtime state information
state = "/apps/rancher/containerd"
[plugins.opt]
  path = "/apps/rancher/k3s/agent/containerd"
[plugins.cri]
  stream_server_address = "127.0.0.1"
  stream_server_port = "10010"
  enable_selinux = false
  sandbox_image = "docker.io/rancher/pause:3.1"

[plugins.cri.containerd]
  disable_snapshot_annotations = true
  snapshotter = "overlayfs"

[plugins.cri.cni]
  bin_dir = "/apps/rancher/k3s/data/3a24132c2ddedfad7f599daf636840e8a14efd70d4992a1b3900d2617ed89893/bin"
  conf_dir = "/apps/rancher/k3s/agent/etc/cni/net.d"

[plugins.cri.containerd.runtimes.runc]
  runtime_type = "io.containerd.runc.v2"

[root@sampleserver containerd]# 

looks like the config is correct but the state property is getting ignored i believe its because it's hardcoded in the links below
config.go:

k3s/pkg/agent/config/config.go

Line 455 in 4d1f9ed

nodeConfig.Containerd.State = "/run/k3s/containerd"

containerd.go:

k3s/pkg/agent/containerd/containerd.go

Line 51 in 88dd601

"--state", cfg.Containerd.State,

Looks like you got a similar issue open for this here: #1358

@brandond do you know of a way to override nodeConfig.Containerd.State that config.go provides to get my desired state directory location? Or in you option should i setup Containerd outside of k3s?

[brandond]

Yeah in that case your best bet is to just run your own containerd and point K3s at the socket.

[BayanAzima]

decided to take another approach while testing on my one node cluster. I noticed that the kublet was running under /var/lib so i decided to try adding --kubelet-arg='root-dir=/apps/rancher/kubelet' to the start command and that seemed to use my /apps/ directory mount!

before kubelet root directory change:

Capacity:
cpu:                80
ephemeral-storage:  18422Mi
hugepages-1Gi:      0
hugepages-2Mi:      0
memory:             527854476Ki
pods:               110

after the change of adding --kubelet-arg='root-dir=/apps/rancher/kubelet':

Capacity:
  cpu:                80
  ephemeral-storage:  10986354Mi
  hugepages-1Gi:      0
  hugepages-2Mi:      0
  memory:             527854476Ki
  pods:               110

[brandond]

Have you tried the k3s --data-dir setting?

[BayanAzima]

yeah my full command is below, seems data-dir is not enough. I believe this since the kublet is reporting on the volume it's installed on, not for certain.

I am having to do this because my kubernetes jenkins agents for some projects had large workspaces and the pods were getting evicted due to disk pressure. I am hoping this fixes it.

cat install_k3s.sh | \
INSTALL_K3S_VERSION="v1.19.4+k3s1" \
K3S_TOKEN='secret' \
sh -s - \
--disable=traefik \
--cluster-cidr 100.100.0.0/18 \
--service-cidr 100.100.64.0/18 \
--write-kubeconfig-mode 644 \
--data-dir /apps/rancher/k3s/ \
--kubelet-arg='root-dir=/apps/rancher/kubelet' \
--cluster-init \
server