Perfomance docs and recommendations

[Anna-Katona]

Is there an existing issue for this?

• I have searched the existing issues

Is your feature request related to a problem?

I have Tetragon installed on a high-load node with more than 10 different tracing policies.
I’m seeing many errors in tetragon_bpf_missed_events_total (ENOSPC), tetragon_observer_ringbuf_queue_events_lost_total, and tetragon_observer_ringbuf_events_lost_total. I tried increasing --rb-queue-size and --rb-size-total, but it doesn’t seem to help and sometimes even makes things worse.

Describe the feature you would like

It would be great to have a documentation page with recommendations for tuning Tetragon performance under high load. There are metrics that indicate drops/errors in the kernel and in user-space queues, and there are multiple configuration options for ring buffers, caches, and queues. However, it’s not obvious which settings should be adjusted to reduce the number of errors/dropped events.

Describe your proposed solution

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[mtardy]

It would be great to have a documentation page with recommendations for tuning Tetragon performance under high load. There are metrics that indicate drops/errors in the kernel and in user-space queues, and there are multiple configuration options for ring buffers, caches, and queues. However, it’s not obvious which settings should be adjusted to reduce the number of errors/dropped events.

I completely agree, that would be great.

From your report the issue is that you get too many missed event from the ring buffer because the userspace side is not reading them fast enough. Indeed, increasing the ring buffer size could help if that's just a temporary problem but if userspace is consuming the events too slowly it won't fix the issue.

If you want to solve your situation here I see two things:

• produce less events: analyze your tracing policy and see if you really need to produce that much events.
• optimize userspace to be capable of processing your flow of events, or at least analyze what's the bottleneck in your situation.

Without having some kind of reproducible setup on our side from the load you are generating and the TracingPolicy and config you use, I'm not sure I can help you here. Maybe you can share more about the setup you have? I think having a setup in which we can investigate those issue would be super interesting for the project to understand what we could optimize indeed.