Processing stopped when we encountered any error in pipeline

[jeyakumar8]

Hi Team,

We are using version 2.2.0 running in Kubernetes pods and have over 100 Kafka partitions. Occasionally, we experience lag building up without any apparent reason, and I don't see any error messages in the logs.

I have verified the following:

• The Splunk endpoint is active.
• There are no resource constraints; both memory and CPU usage look good.
• No issues on the kafka side.
• All the tasks are showing running when hitting curl http://localhost:8083/connectors/<group_id>/status

If anyone has any insights, could you please assist?

Error message
[2024-09-19 16:11:12,333] ERROR WorkerSinkTask{id=} RetriableException from SinkTask: (org.apache.kafka.connect.runtime.WorkerSinkTask) org.apache.kafka.connect.errors.RetriableException: com.splunk.hecclient.HecException: max outstanding events 1000000 have reached, pause the pull for a while Caused by: com.splunk.hecclient.HecException: max outstanding events 1000000 have reached, pause the pull for a while

Thanks in Advance