Repository navigation
Ingest data from Prometheus as Source #1997
Description
Is your feature request related to a problem? Please describe.
Pipeline authors wanted to fetch data to Prometheus
Describe the solution you'd like
Create a source plugin which will fetch data from Prometheus and generate Data Prepper Events. This plugin should be able to handle scale and performance targeted while designing the plugin
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Remote write example: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/README.md
Activity
[-]Ingest data from Prometheus as Source[/-][+]Ingest data from Prometheus as Sink[/+][-]Ingest data from Prometheus as Sink[/-][+]Ingest data to Prometheus as Sink[/+][-]Ingest data to Prometheus as Sink[/-][+]Ingest data from Prometheus as Source[/+]srikanthpadakanti commentedon Feb 11, 2026
Hello @dlvenable One qq:
The GitHub issue says "fetch data from prometheus" which implies pull/scrape model and the reference provided "Remote write example" is Receive/Push model. There are opposite data flow directions. Which interpretation is correct for this open issue? If there is any existing RFC, please point me to that. Thanks.
dlvenable commentedon Feb 17, 2026
@srikanthpadakanti , I believe this issue has not been fully fleshed out. Ultimately, Data Prepper could probably support both approaches depending on the need. However, I think that pull/scrape model would probably be the ideal solution in general here. It would allow Data Prepper to read from Prometheus without having to add any additional configurations to Prometheus beyond what is required for any other pull/scrape reader.
srikanthpadakanti commentedon Apr 3, 2026
@dlvenable Which one you are leaning towards? Polling with Worker Thread (like CloudWatch) or Scheduled Task (like RSS)
srikanthpadakanti commentedon Apr 6, 2026
@dlvenable I am leaning towards the Scheduled Task approach. Prometheus scraping is inherently periodic and each scrape is an independent snapshot with no state or pagination to track between runs.
I am planning to add this to the existing prometheus source plugin so it supports both remote write and scrape in one plugin. The scrape config would be optional.