Description
Describe the bug
When using the file source with a codec configured and tail: false (default), the file is read repeatedly in an infinite loop instead of being read once and stopping.
To Reproduce
Steps to reproduce the behavior:
- Create a simple input file (
input.log):
2024-01-01 INFO line one
2024-01-01 INFO line two
2024-01-01 INFO line three
- Create a pipeline configuration:
test-pipeline:
source:
file:
path: "/path/to/input.log"
record_type: "event"
codec:
newline:
sink:
- file:
path: "/path/to/output.log"- Create
data-prepper-config.yaml:
ssl: false- Run Data Prepper:
./bin/data-prepper /path/to/pipeline.yaml /path/to/data-prepper-config.yaml- Wait 5-10 seconds and check the output file:
wc -l /path/to/output.logExpected behavior
The output file should contain exactly 3 lines (one event per input line). Data Prepper should read the file once and stop processing.
Actual Behavior
The output file grows continuously, with the same 3 events being duplicated every ~500ms. The file source never stops re-reading.
Screenshots
Environment:
- OS: Amazon Linux 2
- Version: Data Prepper 2.16.0-SNAPSHOT (current main branch)
Additional context
- Affects all codecs (
newline,json, etc.) when used with thefilesource in non-tail mode - Not Affected:
filesource without a codec (uses legacyClassicFileStrategypath — reads once correctly)
Root Cause
In FileReader.run(), when !tailMode && codec != null, the method calls readFileWithCodecOneShot() and returns early. However, lastRotationType is never updated from its initial value of RotationType.NO_ROTATION because the rotation check is below the early return.
Metadata
Metadata
Assignees
Labels
Type
Projects
- StatusShow more project fieldsDone
Activity
kkondaka commentedon Jun 23, 2026
@srikanthpadakanti do you want to take a look at this?
srikanthpadakanti commentedon Jun 23, 2026
@kkondaka @dlvenable Please review this. Thanks.
Fix file source infinite re-read in non-tail mode with codec (#6934) (#…
Fix file source infinite re-read in non-tail mode with codec (opensea…