record_modifier BOOL_MAP_LIMIT memory leak

[wiardvanrij]

Bug Report

We run fluentbit on K8s. On two pods from the 100's, we get this error: The number of elements exceeds limit 65535

It is happening on this filter:

    [FILTER]
        Name record_modifier
        Match *
        Remove_key xxx_docker_id
        Remove_key xxx_container_hash
        Remove_key xxx_labels_pod-template-hash
        Remove_key _p

I've looked a little bit at the code:

fluent-bit/plugins/filter_record_modifier/filter_modifier.c

Line 369 in c2b3d47

if (map_num > BOOL_MAP_LIMIT) {

and if I infer this correctly, it should mean I have that many elements in my log body?

To split this problem into two sections;

1. I debugged and looked into the raw logs and could not find any log line that would have caused this error in the first place
2. Regardless, let's assume we do; this is causing a memory leak. The two pods in question surpasses our buffer sizes and limits over time. Clearly indicating that something is staying in memory every time we hit this error.

Potentially:

At a first glance it seems to me that both exit paths in lines 372 and 378 do not dispose of neither the encoder nor the decoder and I'm fairly sure that at least the encoder part will leak the internal msgpack working buffer in this case

I'm unable to provide my full config as it contains many business specific things and we have some company policies. Like-wise as for my raw logs. Hopefully this is sufficient for now.

If anything, I moved this record_modifier into a plain modifier and that solves all our problems.

(we run on v2.2.x, which is old, but I do assume the logic on this part hasn't changed much)