[BUG] RST_STREAM non retryable error with version 2.12.2 #6271
Description
Describe the bug
After upgrading Data Prepper from 2.11.0 to 2.12.2 we get regular (every few seconds) error responses with the sender (e.g. OTel collector):
2025-11-14T15:25:00.935Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "84949179-7201-4a9d-924c-6a76d7982653", "service.name": "otelcol-contrib", "service.version": "0.139.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "not retryable error: Permanent error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR", "dropped_items": 50}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
go.opentelemetry.io/collector/exporter/[email protected]/internal/queue_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume
go.opentelemetry.io/collector/exporter/[email protected]/internal/queuebatch/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue.(*asyncQueue[...]).Start.func1
go.opentelemetry.io/collector/exporter/[email protected]/internal/queue/async_queue.go:49
When downgrading to 2.11.0 it disappears again.
We can't observe any warning/error logs during ingestion directly within Data Prepper.
However, we noticed this warning message during Data Prepper startup which wasn't there before and thought it might have something to do with the grpc errors:
2025-08-13T16:18:40,867 [main] WARN com.linecorp.armeria.common.CommonPools - Failed to register the common worker group as non-blocking for Reactor. Please consider upgrading Reactor to 3.7.0 or newer.
To Reproduce
Steps to reproduce the behavior:
- Start DP version 2.12.2
- Run an OTel collector (latest version)
- Send Data to the OTel collector (e.g. tested with 500 OTel logs/s)
E.g. with telemtrygen:telemetrygen logs --otlp-insecure --duration inf --rate 500 - Observer the error message every few seconds/minutes
Configuration of collector:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
cors:
allowed_origins:
- "*"
processors:
batch:
send_batch_size: 50
timeout: 1s
exporters:
debug:
verbosity: detailed
otlp:
endpoint: <PLACEHOLDER>
tls:
insecure_skip_verify: true
cert_file: "./cert.crt"
key_file: "./key.key"
service:
telemetry:
logs:
level: "warn"
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug, otlp]
Expected behavior
No non retryable error messages / dropped data
Screenshots
Environment (please complete the following information):
- Running Data Prepper on K8s
Additional context