failed to create containerd task: failed to create shim

[zeitler]

Hi

I'm having a problem and I'm not able to find a way to solve it
I'm installing mongodb, with mongodb Community Operator.

Everything goes fine. Already installed it in another cluster.
But now, I'm having

Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/c99d4ec1-2772-4770-9d0a-250c50d3c456/volumes/kubernetes.io~empty-dir/healthstatus" to rootfs at "/var/log/mongodb-mms-automation/healthstatus": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongodb-agent/rootfs/var/log/mongodb-mms-automation/healthstatus: permission denied: unknown

This errors are with EmptyDir volumes.

And I don't find any way to discover whats happening.

Look around witth kubectl events and only info like that.
With ctr utility command also dont found anything.
Already removed the node and add it again to the cluster.
It happens in more than one node

Using rke2 v1.24.10+rke2r1, in Debian 12 Bookworm

Can you help me to find out why

Thanks in advance

[brandond]

Please fill out the issue template when reporting a problem. Since you didn't fill out the issue template, I've converted this to a discussion.

error during container init: error mounting "/var/lib/kubelet/pods/c99d4ec1-2772-4770-9d0a-250c50d3c456/volumes/kubernetes.io~empty-dir/healthstatus" to rootfs at "/var/log/mongodb-mms-automation/healthstatus": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongodb-agent/rootfs/var/log/mongodb-mms-automation/healthstatus: permission denied: unknown

Something is preventing the kubelet from mounting the emptydir volume to that path. Are you using selinux or apparmor on this host? Have you looked at the pod spec to see if there are there conflicting volume mounts?

[zeitler]

Thank you

My thougts were the same.
Selinux it's not used. Apparmor it's not causing the issue. But just in case I've stopped the service and the same thing happens

In /var/log/daemon.log

Jun  4 20:59:39 vm01 systemd[1]: Created slice kubepods-burstable-pod8a28cdbb_a9c8_4231_9817_8124c0182c98.slice - libcontainer container kubepods-burstable-pod8a28cdbb_a9c8_4231_9817_8124c0182c98.slice.
Jun  4 20:59:39 vm01 nfsrahead[3241404]: setting /var/lib/kubelet/pods/8a28cdbb-a9c8-4231-9817-8124c0182c98/volumes/kubernetes.io~nfs/data-volume-mongodb-0 readahead to 128
...
Jun  4 20:59:41 vm01 systemd[1]: run-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-f6a2da1a7ed6465796d6618d9154fc2e61bf7d16692b6e461c60f95187979626-rootfs.mount: Deactivated successfully.
Jun  4 20:59:43 vm01 systemd[1]: var-lib-rancher-rke2-agent-containerd-tmpmounts-containerd\x2dmount3665763629.mount: Deactivated successfully.
Jun  4 20:59:43 vm01 systemd[1]: Started cri-containerd-c8833676da65b2e0a2251eb587acff541ee7c5c1c51532403eb356ec1ed638e4.scope - libcontainer container c8833676da65b2e0a2251eb587acff541ee7c5c1c51532403eb356ec1ed638e4.
Jun  4 20:59:43 vm01 systemd[1]: cri-containerd-c8833676da65b2e0a2251eb587acff541ee7c5c1c51532403eb356ec1ed638e4.scope: Deactivated successfully.
Jun  4 20:59:43 vm01 systemd[1]: run-k3s-containerd-io.containerd.runtime.v2.task-k8s.io-c8833676da65b2e0a2251eb587acff541ee7c5c1c51532403eb356ec1ed638e4-rootfs.mount: Deactivated successfully.

And

==> /var/log/kern.log <==
Jun  4 20:59:40 vm01 kernel: [1736737.722932] IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready
Jun  4 20:59:40 vm01 kernel: [1736737.722997] IPv6: ADDRCONF(NETDEV_CHANGE): cali89ddffcbed3: link becomes ready

Nothing that could tell the issue

Is there a way to increase log in kubelet?

[zeitler]

And tried another deployment for testing with an EmptyDir and worked without issues

[brandond]

Is there another volume mounted somewhere in the pod's /var/log/mongodb-mms-automation/healthstatus path that this empydir is attempting to be mounted into? If so, what kind of volume is it?

[zeitler]

More volumes have errors, but healthstatus as an example:

containers:
- name: mongod:
  volumeMounts:
    - mountPath: /var/log/mongodb-mms-automation/healthstatus
      name: healthstatus

- name: mongodb-agent
  volumeMounts:
    - mountPath: /var/log/mongodb-mms-automation/healthstatus
      name: healthstatus
 
 volumes:
   - emptyDir: {}
     name: healthstatus

The entire volumes section:

- emptyDir: {}
  name: agent-scripts
- name: automation-config
  secret:
    defaultMode: 416
    secretName: mongodb-config
- emptyDir: {}
  name: healthstatus
- emptyDir: {}
  name: hooks
- emptyDir: {}
  name: mongodb-keyfile
- emptyDir: {}
  name: tmp

Other errors of the same kind of the first I've presented:

Warning  Failed     9m30s                   kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/rancher/rke2/agent/containerd/io.containerd.grpc.v1.cri/containers/mongod/volumes/b1198de16015fd027a5a854af778e7a18340f29976fe98e16a1c591bbefe46c8" to rootfs at "/data/db": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongod/rootfs/data/db: permission denied: unknown
Warning  Failed     9m28s                   kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting 
"/var/lib/rancher/rke2/agent/containerd/io.containerd.grpc.v1.cri/containers/mongod/volumes/56772e9110255cc3c0114541bad357283a25964868c7ca6412af63d2c3482b74" to rootfs at "/data/db": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongod/rootfs/data/db: permission denied: unknown

Warning  Failed     9m27s (x2 over 9m29s)   kubelet            Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/var/lib/kubelet/pods/1cb8fdf8-7cb1-48a6-93f9-8417080d0317/volumes/kubernetes.io~empty-dir/healthstatus" to rootfs at "/var/log/mongodb-mms-automation/healthstatus": mkdir /run/k3s/containerd/io.containerd.runtime.v2.task/k8s.io/mongodb-agent/rootfs/var/log/mongodb-mms-automation/healthstatus: permission denied: unknown

[brandond]

Something on your host is returning permission denied when the kubelet attempts to mount those volumes. I don't really have any clue what it might be. You might consider asking on the project of whoever maintains that operator if this is something that they've run into in the past?

[zeitler]

ok

thank you very much