Skip to content

[BUG] File source re-reads file indefinitely when using codec in non-tail mode #6934

Closed

Description

@yavmanis

Describe the bug
When using the file source with a codec configured and tail: false (default), the file is read repeatedly in an infinite loop instead of being read once and stopping.

To Reproduce
Steps to reproduce the behavior:

  1. Create a simple input file (input.log):
2024-01-01 INFO line one
2024-01-01 INFO line two
2024-01-01 INFO line three
  1. Create a pipeline configuration:
test-pipeline:
  source:
    file:
      path: "/path/to/input.log"
      record_type: "event"
      codec:
        newline:
  sink:
    - file:
        path: "/path/to/output.log"
  1. Create data-prepper-config.yaml:
ssl: false
  1. Run Data Prepper:
./bin/data-prepper /path/to/pipeline.yaml /path/to/data-prepper-config.yaml
  1. Wait 5-10 seconds and check the output file:
wc -l /path/to/output.log

Expected behavior
The output file should contain exactly 3 lines (one event per input line). Data Prepper should read the file once and stop processing.

Actual Behavior

The output file grows continuously, with the same 3 events being duplicated every ~500ms. The file source never stops re-reading.

Screenshots

Image

Environment:

  • OS: Amazon Linux 2
  • Version: Data Prepper 2.16.0-SNAPSHOT (current main branch)

Additional context

  • Affects all codecs (newline, json, etc.) when used with the file source in non-tail mode
  • Not Affected: file source without a codec (uses legacy ClassicFileStrategy path — reads once correctly)

Root Cause
In FileReader.run(), when !tailMode && codec != null, the method calls readFileWithCodecOneShot() and returns early. However, lastRotationType is never updated from its initial value of RotationType.NO_ROTATION because the rotation check is below the early return.

Activity

  1. kkondaka commented on Jun 23, 2026

    @kkondaka
    Collaborator

    @srikanthpadakanti do you want to take a look at this?

  2. srikanthpadakanti commented on Jun 23, 2026

    @srikanthpadakanti
    Collaborator

    @kkondaka @dlvenable Please review this. Thanks.

  3. added this to the v2.16 milestone on Jun 30, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

bugSomething isn't working

Type

No type

Projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions