k3s: Change root directory (storing pods) for containerd directory?

[justlucknb]

Hello!

Configuring containerd
K3s will generate config.toml for containerd in /var/lib/rancher/k3s/agent/etc/containerd/config.toml.

For advanced customization for this file you can create another file called config.toml.tmpl in the same directory and it will be used instead.

The config.toml.tmpl will be treated as a Go template file, and the config.Node structure is being passed to the template. [This template](https://github.com/rancher/k3s/blob/master/pkg/agent/templates/templates.go#L16-L32) example on how to use the structure to customize the configuration file.

I need to change root parametr for containerd in k3s server.
I changed /var/lib/rancher/k3s/agent/etc/containerd/config.toml.tmpl to

root = "/mnt/ssd/containerd/k3s/"
[plugins]
  [plugins.opt]
    path = "/var/lib/rancher/k3s/agent/containerd/"
  [plugins.cri]
    stream_server_address = "127.0.0.1"
etc..

based on this documentation:

https://containerd.io/docs/getting-started/

But pods storing at:
/run/k3s/containerd/

How to change root directory (storing pods) for containerd directory?

[chaobei77]

Hello!

I had the same problem,How did you solve it?

[deomorxsy]

I had the same problem,How did you solve it?

containerd has a root and state directory. k3s uses containerd as default high-level container runtime.

• containerd root: /var/lib/rancher/k3s/agent/containerd will be used to store any type of persistent data for containerd. Snapshots, content, metadata for containers and image, as well as any plugin data will be kept in this location. The root is also namespaced for plugins that containerd loads. Each plugin will have its own directory where it stores data. containerd itself does not actually have any persistent data that it needs to store, its functionality comes from the plugins that are loaded.
• containerd state: /run/k3s/containerd will be used to store any type of ephemeral data. Sockets, pids, runtime state, mount points, and other plugin data that must not persist between reboots are stored in this location."

One trick is to create a soft link (symbolic link) with ln for the root directory in the new path, and then pass the new path as the value for the root-dir flag when installing k3s.

MNT_DIR="/target/dir"
K3S_VERSION="<preferred-version>"
# nodefs
#
KUBELET_DIR="${MNT_DIR}/kubelet"
sudo mkdir -p "${KUBELET_DIR}"

# imagefs: containerd has a root and state directory
#
# - https://github.com/containerd/containerd/blob/main/docs/ops.md#base-configuration
#
# containerd root -> /var/lib/rancher/k3s/agent/containerd
#
CONTAINERD_ROOT_DIR_OLD="/var/lib/rancher/k3s/agent"
CONTAINERD_ROOT_DIR_NEW="${MNT_DIR}/containerd-root/containerd"
sudo mkdir -p "${CONTAINERD_ROOT_DIR_OLD}"
sudo mkdir -p "${CONTAINERD_ROOT_DIR_NEW}"
sudo ln -s "${CONTAINERD_ROOT_DIR_NEW}" "${CONTAINERD_ROOT_DIR_OLD}"

# containerd state -> /run/k3s/containerd
#
CONTAINERD_STATE_DIR_OLD="/run/k3s"
CONTAINERD_STATE_DIR_NEW="${MNT_DIR}/containerd-state/containerd"
sudo mkdir -p "${CONTAINERD_STATE_DIR_OLD}"
sudo mkdir -p "${CONTAINERD_STATE_DIR_NEW}"
sudo ln -s "${CONTAINERD_STATE_DIR_NEW}" "${CONTAINERD_STATE_DIR_OLD}"

# pvs -> /var/lib/rancher/k3s/storage
#
PV_DIR_OLD="/var/lib/rancher/k3s"
PV_DIR_NEW="${MNT_DIR}/local-path-provisioner/storage"
sudo mkdir -p "${PV_DIR_OLD}"
sudo mkdir -p "${PV_DIR_NEW}"
sudo ln -s "${PV_DIR_NEW}" "${PV_DIR_OLD}"

I actually read about this on other issue comment in 2023 from this script by @mdrakiburrahman