Skip to content

[BUG] Invalid document version from events is still included in opensearch bulk requests #6601

New issue
New issue
Closed
#6645
Closed
[BUG] Invalid document version from events is still included in opensearch bulk requests#6601
#6645
Labels
bugSomething isn't working
Milestone
v2.15

Description

@graytaylor0
graytaylor0
opened on Mar 3, 2026

Describe the bug
Using external versioning with document_version and document_version_type in the opensearch sink with invalid version fields sends the event to DLQ, but then also includes it in the bulk request.

Expected behavior
Send the event to DLQ, then do not add it to the bulk request:

data-prepper/data-prepper-plugins/opensearch/src/main/java/org/opensearch/dataprepper/plugins/sink/opensearch/OpenSearchSink.java

Line 484 in 5fec579

logFailureForDlqObjects(List.of(createDlqObjectFromEvent(event, indexName, errorMessage)), e);

Screenshots
If applicable, add screenshots to help explain your problem.

Environment (please complete the following information):

  • OS: [e.g. Ubuntu 20.04 LTS]
  • Version [e.g. 22]

Additional context
Add any other context about the problem here.

Activity

  1. graytaylor0
    added
    bugSomething isn't working
    untriaged
    on Mar 3, 2026
  2. github-project-automation
    added this to Data Prepper Tracking Boardon Mar 3, 2026
  3. github-project-automation
    moved this to Unplanned in Data Prepper Tracking Boardon Mar 3, 2026
  4. dlvenable
    removed
    untriaged
    on Mar 3, 2026

  5. dlvenable commented on Mar 3, 2026

    @dlvenable
    dlvenable
    on Mar 3, 2026
    Member

    It looks like we do a pre-validation before the bulk request. Then it still goes into the bulk request.

    I'm going to guess it ends up in the DLQ twice. @graytaylor0 , Is that the case?

  6. graytaylor0 commented on Mar 9, 2026

    @graytaylor0
    graytaylor0
    on Mar 9, 2026
    MemberAuthor

    @dlvenable Yes it goes into the DLQ twice, but the bigger issue is that opensearch had a bug that caused one invalid version document to fail the entire bulk request so the entire bulk request gets sent to DLQ

  7. Keyur-S-Patel
    added 4 commits that reference this issue on Mar 17, 2026

    Fix invalid document version events still included in bulk requests (o…

    cea3661

    Fix invalid document version events still included in bulk requests (o…

    c935360

    Fix invalid document version events still included in bulk requests (o…

    506ee95

    Fix invalid document version events still included in bulk requests (o…

    63d290a
  8. dlvenable
    added this to the v2.15 milestone on Mar 18, 2026
  9. oeyh
    added a commit that references this issue on Mar 18, 2026

    Fix invalid document version events still included in bulk requests (#…

    d4bbcd9
  10. oeyh
    closed this as completedin #6645on Mar 18, 2026
  11. github-project-automation
    moved this from Unplanned to Done in Data Prepper Tracking Boardon Mar 18, 2026
  12. Keyur-S-Patel
    added a commit that references this issue on Apr 16, 2026

    Fix invalid document version events still included in bulk requests (o…

    6d81b89
  13. oeyh
    added a commit that references this issue on Apr 18, 2026

    Fix invalid document version events still included in bulk requests (#…

    4865dce
  14. kkondaka
    added 2 commits that reference this issue on Jul 1, 2026

    Fix invalid document version events still included in bulk requests (o…

    785a582

    Fix invalid document version events still included in bulk requests (o…

    e24bb6a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions