Add support for spec.hostSelector

[tpapagian]

This PR adds support for spec.hostSelector to enable users to write policies that are applied only to host workloads and/or pod workloads (with additional filtering for the containers if needed).

The syntax is:

spec:
  hostSelector: {}
  podSelector: {}
  containerSelector: {}

and the previous example will match all workloads either in the host or inside pods without any filtering. This is also the default (i.e. if the user does not define any of those in a tracing policy).

If we need to match only host workloads we need to have something like:

spec:
  hostSelector: {}
  podSelector: null
  containerSelector: null

For now we only support 2 types of hostSelector. These are the {} (everything) and null (nothing). Specific filters are not supported yet.

In order to make that work, we need to make some changes in the behaviour of the podSelector. Currently, podSelector: {} and podSelector: null are exactly the same thing and will match all workloads (including host workloads). Based on this behaviour, now we don't have a way to match on all pod workloads (excluding host workloads).

In this PR we change that behaviour. Now podSelector: {} will match all pod workloads and podSelector: null will not match any pod workloads. The behaviour of containerSelector is similar, but what actually does is a second level filtering on the podSelector. For this reason:

spec:
  hostSelector: ~
  podSelector: ~
  containerSelector:
    matchExpressions:
      - key: name
        operator: In
        values:
        - ubuntu

will not match anything as podSelector is null. We need to have something like

spec:
  hostSelector: ~
  podSelector: {}
  containerSelector:
    matchExpressions:
      - key: name
        operator: In
        values:
        - ubuntu

to filter containers.

This PR also provides the ability to make hostSelector, podSelector, and containerSelector equals to null which was not possible before.

Implementation

In order to make that work, this PR adds one more entry in policy_filter_maps with key equals to ALL_PODS_POLICY_ID (UINT32_MAX). The inner map of that entry contains all cgroup_ids of all pods in the system. If a cgroup_id is part of that set, this means that it is a pod workload. If not this means that this is part of the host workload. Additionally, each policy contains an invalid cgroup_id entry with value HOST_SELECTOR_MODE (UINT64_MAX) if the policy itself cares about host workloads.

Then we update policy_filter_check to check those as well. More specifically what policy_filter_check does is:

1. check if our cgroup_id match on the policy build from podSelector/containerSelector
   1. return true if we match
   2. continue to check if we care about host workloads
2. check if the policy cares also for host workloads
   1. return false if not
   2. continue to check if our cgroup_id is a host workload
3. check if our cgroup_id is part of ALL_PODS_POLICY_ID
   1. return true if not (this means that out cgroup_id is not a pod workload --> so this is a host workload)
   2. return false if it is (this means that out cgroup_id is a pod workload)

Any updates needed for the policy_filter_maps are handled from the agent in a similar way to podSelector/containerSelector.

Testing

All tests related to the policyfiler have been updated to reflect those changes. Some additional tests have been added to check the hostSelector explicitly.

Limitations

This approach has similar issues with Kubernetes Identity Aware Policies and more specifically if we rely on the k8s API server, there may be a race on when we add a cgroup_id to our maps and when the pod actually starts. The hostSelector has similar issues. All of these issues can be resolved by using Runtime Hooks instead of the K8s API server.

Alternatives

Another way to do that is to use bpf_current_task_under_cgroup or bpf_get_current_ancestor_cgroup_id eBPF helpers. This allows us to check if a cgroup_id is somewhere under a specific cgroup subtree. If this specific cgroup subtree is where k8s adds new pods, then we can achieve the same results. The challenge here is to find the cgroup_id that k8s use to keep all pods. In most of the cases this is /sys/fs/cgroup/kubepods.slice/ but this is not always the case in all deployments. It seems that there are ways to get that inside Tetragon but there may be a bit complicated.

For this reason, we are starting with the approach proposed in this PR. It works in all cases, although it introduces a small increase in memory usage. At this point, the impact does not seem significant enough to justify a more complex design. We can always improve it in future PRs if needed.

[netlify]

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	c6afde5
🔍 Latest deploy log	https://app.netlify.com/projects/tetragon/deploys/69d50701aab13e0008a20fbc
😎 Deploy Preview	https://deploy-preview-4814--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.