Sandbox/Pause containers should run on a configurable cpuset to avoid interfering with low-jitter workloads

[james-masson]

What is the problem you're trying to solve

Consider a system set up with low-jitter tolerance workloads isolated with Cpusets - as is typically done with K8s cpu-manager static policy.

https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/

When correctly done, low-jitter workloads will have cpusets that do not overlap with anything else - apart from pause/sandbox containers. Containerd is not currently able to isolate pause/sandbox containers from "real" workloads, they just inherit the full cpuset of the system.

Here's example output from a well set up system showing pid/cgroup/command as well as the cpu mask from /proc/<pid>/status

eg. a pause container

  16710 0::/kubepods.slice/kubepods pause
Cpus_allowed_list:      0-35

compare to an OS process

    777 0::/system.slice/systemd-jo systemd-journal
Cpus_allowed_list:      0-1,18-19

compare to a normal K8s container

   4142 0::/kubepods.slice/kubepods aws-efs-csi-dri
Cpus_allowed_list:      0-1,11-19,28-35

compared to a low-jitter K8s container

  16845 0::/kubepods.slice/kubepods aeronmd_dpdk
Cpus_allowed_list:      7

New workloads will generate new pause/sandbox containers and potentially disrupt the low-jitter container, because of it's lax CPU mask/cpuset

Describe the solution you'd like

A flag to put pause/sandbox containers on a specific defined cpuset , so they're away from anything that cares about low jitter.

Please see this similar feature on CRIO - cri-o/cri-o#4459

Additional context

Here's an upstream K8s discussion for context: kubernetes/kubernetes#99895