Repository navigation
Open Source Prometheus Support for prometheus-sink #6594
Description
Description
The prometheus-sink plugin is currently hardcoded for AWS Managed Prometheus (AMP), making it impossible to use with open source Prometheus instances. Additionally, the README documents many features that are not actually implemented.
Issue 1: Open Source Prometheus (No Auth) Is Not Supported
The aws configuration block is annotated with @NotNull, requiring it for every pipeline. This means:
- HTTP URLs (
http://) are rejected by theisValidConfig()validation — onlyhttps://is allowed - A SigV4 signer is always created, which fails without AWS credentials
- The
x-amz-content-sha256header is unconditionally added to every request, which vanilla Prometheus does not expect - The
Content-Encodingheader is sent as"SNAPPY"(uppercase, from Java enumtoString()) instead of"snappy"(lowercase), which Prometheus rejects
Expected Behavior
Users should be able to send metrics to an open source Prometheus Remote Write endpoint with a simple config like:
sink:
- prometheus:
url: "http://localhost:9090/api/v1/write"Affected Files
PrometheusSinkConfiguration.java—@NotNullonawsConfig,isValidConfig()rejectshttp://PrometheusHttpSender.java— unconditionalx-amz-content-sha256header, uppercaseContent-Encoding
Issue 2: README Documents Features That Are Not Implemented
The current README (data-prepper-plugins/prometheus-sink/README.md) documents configuration options and usage examples for features that have config classes defined but are never wired up in the actual request path.
| Documented Feature | Status |
|---|---|
HTTP Basic authentication (authentication.http_basic) |
Config classes exist, never instantiated |
Bearer token / OAuth authentication (authentication.bearer_token) |
Config classes exist, never instantiated |
Proxy support (proxy) |
Not implemented |
Custom headers (custom_header) |
Not implemented |
HTTP method (http_method) |
Not implemented (always POST) |
SSL/TLS options (insecure, insecure_skip_verify, ssl_certificate_file, etc.) |
Not implemented |
DLQ file/plugin (dlq_file, dlq) |
Not implemented |
Buffer type (buffer_type) |
Not implemented |
sigv4 toggle in AWS config |
Not implemented (SigV4 is always on when aws is present) |
Note: The handler classes (
BasicAuthPrometheusSinkHandler,BearerTokenAuthPrometheusSinkHandler,MultiAuthPrometheusSinkHandler) exist but use Apache HttpClient 5, while the actual sender (PrometheusHttpSender) uses Armeria. They are never called.
Expected Behavior
The README should only document features that are actually functional, with unimplemented features clearly marked or removed.
Environment
- Data Prepper version: 2.15 (main branch)
- Prometheus: v2.x / v3.x with
--web.enable-remote-write-receiver
Activity
[-][BUG] Open Source Prometheus Support[/-][+][BUG] Open Source Prometheus Support for prometheus-sink[/+][-][BUG] Open Source Prometheus Support for prometheus-sink[/-][+]Open Source Prometheus Support for prometheus-sink[/+]