Repository navigation
[BUG] RST_STREAM non retryable error with version 2.12.2 #6271
Description
Describe the bug
After upgrading Data Prepper from 2.11.0 to 2.12.2 we get regular (every few seconds) error responses with the sender (e.g. OTel collector):
2025-11-14T15:25:00.935Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "84949179-7201-4a9d-924c-6a76d7982653", "service.name": "otelcol-contrib", "service.version": "0.139.0"}, "otelcol.component.id": "otlp", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "not retryable error: Permanent error: rpc error: code = Internal desc = stream terminated by RST_STREAM with error code: INTERNAL_ERROR", "dropped_items": 50}
go.opentelemetry.io/collector/exporter/exporterhelper/internal.NewQueueSender.func1
go.opentelemetry.io/collector/exporter/exporterhelper@v0.139.0/internal/queue_sender.go:49
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queuebatch.(*disabledBatcher[...]).Consume
go.opentelemetry.io/collector/exporter/exporterhelper@v0.139.0/internal/queuebatch/disabled_batcher.go:23
go.opentelemetry.io/collector/exporter/exporterhelper/internal/queue.(*asyncQueue[...]).Start.func1
go.opentelemetry.io/collector/exporter/exporterhelper@v0.139.0/internal/queue/async_queue.go:49
When downgrading to 2.11.0 it disappears again.
We can't observe any warning/error logs during ingestion directly within Data Prepper.
However, we noticed this warning message during Data Prepper startup which wasn't there before and thought it might have something to do with the grpc errors:
2025-08-13T16:18:40,867 [main] WARN com.linecorp.armeria.common.CommonPools - Failed to register the common worker group as non-blocking for Reactor. Please consider upgrading Reactor to 3.7.0 or newer.
To Reproduce
Steps to reproduce the behavior:
- Start DP version 2.12.2
- Run an OTel collector (latest version)
- Send Data to the OTel collector (e.g. tested with 500 OTel logs/s)
E.g. with telemtrygen:telemetrygen logs --otlp-insecure --duration inf --rate 500 - Observer the error message every few seconds/minutes
Configuration of collector:
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
cors:
allowed_origins:
- "*"
processors:
batch:
send_batch_size: 50
timeout: 1s
exporters:
debug:
verbosity: detailed
otlp:
endpoint: <PLACEHOLDER>
tls:
insecure_skip_verify: true
cert_file: "./cert.crt"
key_file: "./key.key"
service:
telemetry:
logs:
level: "warn"
pipelines:
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug, otlp]
Expected behavior
No non retryable error messages / dropped data
Screenshots
Environment (please complete the following information):
- Running Data Prepper on K8s
Additional context
Activity
JannikBrand commentedon Nov 17, 2025
Couldn't reproduce this with docker compose yet..
In might be related to something within our environment => 2 DP Pods in K8s; behind ingress-nginx; using gRPC
JannikBrand commentedon Nov 28, 2025
also the case for 2.13.0
KarstenSchnitter commentedon Dec 1, 2025
This looks like a conflict in the
reactor-coreversions. These dependencies are introduced by Armeria (causing the warning), Spring and the Azure libraries. Spring and Microsoft Office 365 Source remain on rather old versions which seems to be used. Spring itself does not pull the dependency with spring-core, spring-context, or spring-web. spring-webflux would use reactor-core:3.4.41, which is the version used in Data Prepper.kkondaka commentedon Dec 2, 2025
@JannikBrand @KarstenSchnitter would either of you able to work on this?
KarstenSchnitter commentedon Dec 5, 2025
We are looking into it. We are not sure, what is causing the issues, yet.
JannikBrand commentedon Dec 8, 2025
We identified that the root cause of this issue is the current armeria version (1.32.5), which is affected by a couple of bugs that are mentioned to be fixed within the 1.33.0 release notes. Namely:
It seems that versions 1.34.0 and 1.33.2 also had related bug fixes that might be affecting the current version:
I built a custom DP version with armeria version 1.34.1 and necessary changes due to (transitive) dependency updates and could verify that this resolves the issues in our environment.
I will align with @KarstenSchnitter tomorrow on providing the fix.
Update Ameria to 1.34.1
Update Ameria to 1.34.1
Update Ameria to 1.34.1
Update Ameria to 1.34.1
Update Ameria to 1.34.1
Update Ameria to 1.34.1
Update Ameria to 1.34.2
Update Armeria to 1.32.6
Update Armeria to 1.32.6
Update Armeria to 1.32.6
Update Armeria to 1.32.6 (#6343)
Update Armeria to 1.32.6 (opensearch-project#6343)
Update Armeria to 1.32.6 (#6343) (#6820)
Update Armeria to 1.32.6 (opensearch-project#6343)