SQL input plugin

[mudim]

Hi all, hope all well. Looking for some clarification around sql input plugin, in context of logs.
https://github.com/fluent/fluent-plugin-sql

Documentation is clear on how the state is maintained so that we don't fetch same records on restarts. However, when deployed as daemonset, want to understand how the state is replicated across the plugins (one one each pod). Because, if the state is not replicated, wouldn't each of the input query same data there by duplicating log records?

Cheers

[daipom]

I'm not familiar with k8s, but if you collect data outside of the instance, then it would be necessary to put the state file outside as well, I think.

[manusabu11]

Hi all, hope all well. Looking for some clarification around sql input plugin, in context of logs. https://github.com/fluent/fluent-plugin-sql

Documentation is clear on how the state is maintained so that we don't fetch same records on restarts. However, when deployed as daemonset, want to understand how the state is replicated across the plugins (one one each pod). Because, if the state is not replicated, wouldn't each of the input query same data there by duplicating log records?

Cheers

++++++++++
Have you got a solution for this? I am having the same issue as state pos file is on each node. Other than a shared location of pos file ( not tried) do we have any other solution from plugin point of side

+++!++!++

[manusabu11]

I am doing a workaround here as logs are duplicating by creating a new Helm release with deployment for that particular task, by executing it from one node by node selector