You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[S3-SQS Source] Make shutdown timeout configurable to prevent message loss during scale-down #6442
Is your feature request related to a problem? Please describe.
When OpenSearch Ingestion pipelines using the S3-SQS source scale down, OCUs are terminated with a hardcoded 30-second shutdown timeout. If S3 objects cannot be fully processed within this window, in-flight SQS messages are interrupted, causing:
Message visibility timeout expiration
Messages being re-queued as "old messages" (age: 3,500-3,600 seconds)
Duplicate processing and data inconsistencies
Increased processing costs due to repeated work
This is particularly problematic for pipelines processing large S3 objects or complex transformations requiring more than 30 seconds per message.
This is a good approach and is certainly quite simple to implement. But, I also think we may be able to have a more dynamic shutdown time. And we should be sure that we haven't closed the client until after shutdown.
We probably need a way to tell sources that the pipeline has shutdown as well. The current mechanism to mention that it is shutting down.
Activity
dlvenable commentedon Feb 3, 2026
This is a good approach and is certainly quite simple to implement. But, I also think we may be able to have a more dynamic shutdown time. And we should be sure that we haven't closed the client until after shutdown.
We probably need a way to tell sources that the pipeline has shutdown as well. The current mechanism to mention that it is shutting down.
LeilaMoussa commentedon Feb 5, 2026
I would like to open a PR for this issue.
LeilaMoussa commentedon Feb 5, 2026
Opened PR #6489