Skip to content

[S3-SQS Source] Make shutdown timeout configurable to prevent message loss during scale-down #6442

Closed

Description

@Davidding4718

Is your feature request related to a problem? Please describe.
When OpenSearch Ingestion pipelines using the S3-SQS source scale down, OCUs are terminated with a hardcoded 30-second shutdown timeout. If S3 objects cannot be fully processed within this window, in-flight SQS messages are interrupted, causing:

  • Message visibility timeout expiration
  • Messages being re-queued as "old messages" (age: 3,500-3,600 seconds)
  • Duplicate processing and data inconsistencies
  • Increased processing costs due to repeated work

This is particularly problematic for pipelines processing large S3 objects or complex transformations requiring more than 30 seconds per message.

The hardcoded timeout is defined in SqsService.java:

Describe the solution you'd like
Add a configurable shutdown_timeout parameter to the S3-SQS source configuration:

source:
  s3:
    sqs:
      queue_url: "https://sqs.region.amazonaws.com/account/queue"
      shutdown_timeout: 300  # seconds, default: 30

This allows users to configure the timeout based on their workload characteristics.

Describe alternatives you've considered (Optional)
N/A

Additional context
N/A

Activity

  1. dlvenable commented on Feb 3, 2026

    @dlvenable
    Member

    This is a good approach and is certainly quite simple to implement. But, I also think we may be able to have a more dynamic shutdown time. And we should be sure that we haven't closed the client until after shutdown.

    We probably need a way to tell sources that the pipeline has shutdown as well. The current mechanism to mention that it is shutting down.

  2. LeilaMoussa commented on Feb 5, 2026

    @LeilaMoussa
    Contributor

    I would like to open a PR for this issue.

  3. LeilaMoussa commented on Feb 5, 2026

    @LeilaMoussa
    Contributor

    Opened PR #6489

  4. added this to the v2.14 milestone on Feb 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions