Monitoring containers in a specific k8s namespace

[Rafaelyot]

Hello, I am using your tool in a project, and I want to be able to only monitor and use the capture module only on the containers in a specific k8s namespace. Is this possible?

[rafaeldtinoco]

@Rafaelyot we have a distinction between capture and trace. Capture is the feature where we capture artifacts (written files, network flows, etc) and is used mainly when tracee is being used as a "introspection" tool (not a runtime detection tool).

You are likely talking about "tracing containers of a specific k8s namespace" as filtering mechanism, then we have this bug: #3711 opened that you can follow. I'll close this discussion so we can focus in that bug for this issue. Feel free to comment again here if needed, but you may follow the resolution for this on that issue.

Cheers!

[Rafaelyot]

@rafaeldtinoco I already implemented a k8s namespace scope filtering, aka, I am able to trace containers of a specific k8s namespace, however, containers deployed after the tracee execution are not captured. What is the code block responsible for tracing newly created containers? Or, what is the code block where the "--scope container=new" is implemented? Maybe I can join both features and fulfill my requirements.

[rafaeldtinoco]

I need you to clarify some terms you used before I can help.

"I already implemented a k8s namespace scope filter" -> You mean you changed some code ? Which part ? To do what ?

"Im able to trace containers of a specific k8s namespace" -> What is the command line or policy that you are using ?

"What is the code block responsible for tracing newly created containers" -> It is not a code block, its a big logic involving eBPF hooks bringing information about cgroup directories, and working as triggers for userland logic talking to container runtime sockets and/or reading files.

Maybe you can be a bit more prolix in what you are doing, showing examples, then I can help you more.

[Rafaelyot]

Yes, I changed the code base to enable filtering by some k8s filters; in my version, I can do "--scope container=namespace:default" (this is probably not the optimal nor correct solution), and this command will only trace containers in the default namespace; however new containers are left apart. All of my changes are here (merged with the version 0.18).
Regarding the mentioned eBPF hooks, in my understanding tracee creates a copy of the "/sys/fs/cgroup/cpuset" in the /tmp folder. Then it attaches some ebpf program to track some changes (creation and deletion of containers) to trace containers deployed after the execution of the tracee. So I want to change some code logic to only include the containers in a specific namespace and not all of them.
Basically, I want to be able to trace every container in a specific namespace, including new containers.

[rafaeldtinoco]

Ok, I'll rebase your code and open a draft PR so I can point things out better, no worries. There is some potential (for example, you added timestamps to captured file, this would solve #2888, etc).

I'll rebase the code and ping you in the draft PR just so we can chat on top of a single DIFF and, perhaps, I can help you with what you need.

Thanks!

[Rafaelyot]

Thank you for all the help!

[rafaeldtinoco]

I've squashed all your commits and will comment (when time allows) parts of it and then we can clarify where things go (and what is going on in your attempt).

The container package needs rework (specially related to encapsulation).

I think you did correct calling enrichment here for already existing containers (specially because they were already mapped by Populate() called right after containers.New().

Maybe we can enrich inside the object (Populate() could call enriching already, for example). This way we would avoid copying data (about existing containers) out of the instance and then feeding to a method inside it again.

Again, the Containers package/types aren't fully encapsulated, so its not your fault.

[rafaeldtinoco]

This is a nice optimization:

And re-reading this code now, there are many others we can have there :.

[rafaeldtinoco]

This change is ok from your perspective. I really dont like the fact that our pipelines (for events, capturing, network capturing and, more recently, control events) have different formats/protocols. In this case, for example, I think the timestamp should have come with the event. We should have at least a default set of fields for all pipelines (or re-think them in the future).

[rafaeldtinoco]

Before we move on, for tracing purposes the correct way of capturing events from a single container/pod would be:

$ sudo ./dist/tracee --install-path /tmp/tracee --cache cache-type=mem --cache mem-cache-size=512 --output option:sort-events --output option:parse-argumen
ts --output json --scope container=new --events uname --events uname.context.containerName="bleh"

You will see that each field of the "event context" is considered at the eventCtxFilter:

and there is a podName for your purposes.

---

Now we have 2 questions:

1. Should we capture only events being filtered or should tracee capture all events ? That is probably what you are trying to solve.

Currently we cannot define capture events in the policies (we could capture per policy, so the files could be arranged by capture_dir/policy_number/artifact_name, for example).

Nadav said about "capture events". For now, capture events are internal to tracee only AND they couldn't be filtered (read/write) since they don't have the same event context as regular events (and podName is part of an event context).

Currently the filter for captures is on eBPF side but is independent of event filters/policies:

2. Userland filters versus Kernel filters:

All the filters that are BPF* are filters done in the kernel/ebpf side. In your case, you want an event ContextFilter, also applied to captures, and that is only implemented in userland:

This is good and bad.

Bad because you cannot tell the kernel NOT TO SEND contents from reads and writes that you are not interested (you have a read/write eBPF perfbuffer that is going to be filled with events you wont print/write to artifacts).

Good because it is easier to implement. IF the capture events had event context (the Exec capture has, right ? reads and writes don't), you could simply check for the Filter() function output and take an action.

Issues:

1. Captures aren't uniform (so no event context filter for all of them).
2. Existing capture filters dont have a clue about policies or the filter types implementations.

So Summarizing (and I might write more later about it). For the capture feature:

• Write: sent by an eBPF program tailcall.
• Read: sent by an eBPF program tailcall.

To understand "a bit", check core.go event CaptureFileWrite. It has some tailcalls. You can read about eBPF tailcalls if needed. There is one called send_bin. This means that if someone calls the index 3 of "prog_array" map, it will execute the "send_bin" eBPF program:

SEC("kprobe/send_bin")
int BPF_KPROBE(send_bin)
{
    return send_bin_helper(ctx, &prog_array, TAIL_SEND_BIN);
}

reading send_bin_helper:

You can see that we're picking the cgroup_id for the current task and saving in a buffer that will be submitted to userland (that will be the cgroup_id field decoded in the perfbuffer responsible for sending file contents). See ? We dont have an event context for these events, nor a proper filter.

Since you have the "cgroupId" you could "filter out" somewhere in the code, like you're doing, picking cgroupId, then cgroupInfo struct and checking for the podName, for example.

But the scope=container and --events uname.context.containerName="bleh" is totally disconnected to the capture data being submitted (for both read/write).

I think "this answers a bit" your question about new events showing up or not for the capture events.

I still need to read the rest of your filter code to comment more, anyway we have a baseline to talk I believe.