Repository navigation
[BUG] Error Processing Files Larger Than 2GB Due to Integer Overflow in Data Prepper #5276
Description
Describe the bug
When processing files larger than 2GB, the system throws an error indicating that the numeric value is out of range for an integer.
To Reproduce
Steps to reproduce the behavior:
- Upload a file larger than 2GB to the specified S3 bucket.
- Ensure that the S3EventBridgeNotification is configured to trigger when a new file is uploaded.
- Once the Data Prepper pipeline is triggered, the error shows up in the logs.
Expected behavior
The system should handle files larger than 2GB without throwing an error.
Additional context
404 [s3-source-sqs-1] ERROR org.opensearch.dataprepper.plugins.source.s3.parser.S3EventBridgeNotificationParser - SQS message with message ID:8947a26b-2500-4859-bfd1-05353e0fc232 has invalid body which cannot be parsed into EventBridgeNotification. Numeric value (2409161669) out of range of int (-2147483648 - 2147483647)
at [Source: REDACTED (`StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION` disabled); line: 1, column: 523] (through reference chain: org.opensearch.dataprepper.plugins.source.s3.S3EventBridgeNotification["detail"]->org.opensearch.dataprepper.plugins.source.s3.S3EventBridgeNotification$Detail["object"]->org.opensearch.dataprepper.plugins.source.s3.S3EventBridgeNotification$Object["size"])
Reacted by Steffen Pingel
Activity
dblock commentedon Jan 6, 2025
[Catch All Triage - 1, 2, 3, 4, 5, 6]
dlvenable commentedon Feb 3, 2025
@parastoosln , Using an
inthere certainly seems wrong. Would you be able to make the change tolong?parastoosln commentedon Feb 11, 2025
No, as an alternative solution, we reduced the file size.
LeilaMoussa commentedon Feb 9, 2026
I will open a PR.