Releases: prometheus/prometheus
Release list
3.14.0 / 2026-08-17
- [CHANGE] API: Deprecate the
statsquery parameter of/api/v1/queryand/api/v1/query_rangefor values other thantrueandall. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 - [CHANGE] API:
/api/v1/status/confignow correctly showsseparator: ""andreplacement: ""in relabel configs when explicitly set to empty, instead of omitting them. #18653 - [CHANGE] Discovery/Hetzner: Drop the
__meta_hetzner_datacenterlabel forhcloudtargets, following its removal from the Hetzner Cloud API. #19269 - [CHANGE] PromQL: Enable duration expressions by default. The
promql-duration-exprfeature flag is now a no-op. #19033 - [CHANGE] PromQL: Promote
first_over_timeto stable. It no longer requires thepromql-experimental-functionsfeature flag. #19093 - [FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (
oci_sd_configs). #18919 - [FEATURE] PromQL: Add experimental
start_timestamp(instant-vector)function returning the start timestamp of each sample in the given vector. Requires theuse-start-timestampsfeature flag. #19089 - [FEATURE] PromQL: Allow
rate()andincrease()to use start timestamps as an alternative for rate extrapolation. Hidden behind theuse-start-timestampsfeature flag. #18619 - [FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the
histograms-st-encodingfeature flag. #18609 - [ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g.
k8s.pod.nameandk8s_pod_nameboth becomek8s_pod_name), and expose theprometheus_api_otlp_translation_warnings_totalcounter labelled bycategoryto track such warnings. #18957 - [ENHANCEMENT] Promtool: Add
--remote-write.pathflag topush metricsfor backends that use a non-default remote-write endpoint. #19086 - [ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903
- [ENHANCEMENT] TSDB: Add
prometheus_tsdb_head_native_histogram_seriesandprometheus_tsdb_head_native_histogram_bucketsgauges tracking the number of native histogram series and buckets in the head. #19170 - [ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (
step(),range(),min_of(),max_of()) in range selectors and subqueries. #18625 - [ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706
- [ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181
- [PERF] Speed up regex label matchers matching a set of literal values (e.g.
{job=~"foo|bar|baz"}). #18833 - [PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470
- [PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143
- [PERF] Scrape: Reduce native histogram scrape parsing allocations by ~49%. #19282
- [PERF] TSDB: Speed up queries on series with many in-memory chunks. #18300
- [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
- [BUGFIX] Discovery/AWS: Stop
promtool check configfrom making AWS metadata service (IMDS) network calls when theregionfield is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037 - [BUGFIX] Discovery/Docker: Set a request timeout for
docker_sdanddockerswarm_sdonunix,npipe, andtcphosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237 - [BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
- [BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778
- [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
- [BUGFIX] PromQL: Fix
mad_over_timereturning 0 instead ofNaNwhen the range contains aNaNsample. #19040 - [BUGFIX] Promtool: Accept
--enable-feature=promql-binop-fill-modifiersincheck rules, which previously rejected validfill()/fill_left()/fill_right()expressions. #19153 - [BUGFIX] Remote write: Respect the
AZURE_FEDERATED_TOKEN_FILEenvironment variable for workload identity authentication instead of hardcoding the token file path. #18973 - [BUGFIX] Rules: Clean up stale
rule_group_last_rule_duration_sum_secondsandrule_group_last_restore_duration_secondsseries when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing/metricscardinality over time. #19107 - [BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
- [BUGFIX] TSDB: Fix silent data loss and potential crash loop when
stale_series_compaction_thresholdis used in the config file. #19016 - [BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted. #19016
- [BUGFIX] TSDB: Fix
prometheus_tsdb_head_stale_seriesover-counting and early eviction of series that change between float, integer histogram, and float histogram sample types. #19183 - [BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
- [BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
- [BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
- [BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart. #19202
- [BUGFIX] TSDB: Surface query errors that were previously silently discarded. #19120
- [BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with
--enable-feature=xor2-encodingcould silently revert to XOR after compaction. #19145 - [BUGFIX] UI: Show the delete-series form on the TSDB Status page when
--web.enable-admin-apiis enabled. #19025
3.14.0-rc.0 / 2026-08-06
- [CHANGE] API: Deprecate the
statsquery parameter of/api/v1/queryand/api/v1/query_rangefor values other thantrueandall. Other values still enable basic statistics but now return a deprecation warning; they will be rejected in the next major release. #19124 - [CHANGE] API:
/api/v1/status/confignow correctly showsseparator: ""andreplacement: ""in relabel configs when explicitly set to empty, instead of omitting them. #18653 - [CHANGE] Discovery/Hetzner: Drop the
__meta_hetzner_datacenterlabel forhcloudtargets, following its removal from the Hetzner Cloud API. #19269 - [CHANGE] PromQL: Enable duration expressions by default. The
promql-duration-exprfeature flag is now a no-op. #19033 - [CHANGE] PromQL: Promote
first_over_timeto stable. It no longer requires thepromql-experimental-functionsfeature flag. #19093 - [FEATURE] Discovery: Add Oracle Cloud Infrastructure compute service discovery (
oci_sd_configs). #18919 - [FEATURE] PromQL: Add experimental
start_timestamp(instant-vector)function returning the start timestamp of each sample in the given vector. Requires theuse-start-timestampsfeature flag. #19089 - [FEATURE] PromQL: Allow
rate()andincrease()to use start timestamps as an alternative for rate extrapolation. Hidden behind theuse-start-timestampsfeature flag. #18619 - [FEATURE] TSDB: Add experimental support for encoding start timestamps in histograms and float histograms. Hidden behind the
histograms-st-encodingfeature flag. #18609 - [ENHANCEMENT] OTLP: Emit a warning when OTLP attribute names collide into the same Prometheus label after sanitization (e.g.
k8s.pod.nameandk8s_pod_nameboth becomek8s_pod_name), and expose theprometheus_api_otlp_translation_warnings_totalcounter labelled bycategoryto track such warnings. #18957 - [ENHANCEMENT] Promtool: Add
--remote-write.pathflag topush metricsfor backends that use a non-default remote-write endpoint. #19086 - [ENHANCEMENT] Remote write: Forward histogram start timestamps in the remote write V2 protocol. #18903
- [ENHANCEMENT] TSDB: Add
prometheus_tsdb_head_native_histogram_seriesandprometheus_tsdb_head_native_histogram_bucketsgauges tracking the number of native histogram series and buckets in the head. #19170 - [ENHANCEMENT] UI: Add syntax highlighting, autocompletion, and linting for PromQL duration expressions (
step(),range(),min_of(),max_of()) in range selectors and subqueries. #18625 - [ENHANCEMENT] UI: Add copy button next to rule names on the Rules and Alerts pages. #18706
- [ENHANCEMENT] UI: Improve rule group title contrast on the Rules page. #19181
- [PERF] Speed up regex label matchers matching a set of literal values (e.g.
{job=~"foo|bar|baz"}). #18833 - [PERF] Remote read: Improve remote read throughput by removing unnecessary per-write flushing. #18470
- [PERF] Scrape: Parse text and OpenMetrics formats without recursion, preventing stack overflow from deeply nested or malicious exposition input. #19143
- [PERF] Scrape: Reduce native histogram scrape parsing allocations by ~49%. #19282
- [PERF] TSDB: Speed up queries on series with many in-memory chunks. #18300
- [BUGFIX] Alerting: Fix 100% CPU usage on shutdown that could delay graceful shutdown and trigger timeout-based kills. #17859
- [BUGFIX] Discovery/AWS: Stop
promtool check configfrom making AWS metadata service (IMDS) network calls when theregionfield is omitted in EC2, ECS, RDS, MSK, ElastiCache, and Lightsail service discovery configs. #19037 - [BUGFIX] Discovery/Docker: Set a request timeout for
docker_sdanddockerswarm_sdonunix,npipe, andtcphosts. Previously an unresponsive daemon could freeze discovery indefinitely, silently pinning targets to a stale snapshot. #19237 - [BUGFIX] Discovery/Docker: Fix panic in Docker Swarm service discovery when a service runs as a plugin or network-attachment. #19102
- [BUGFIX] Discovery/Docker: Fix discovery of IPv6-only containers. #18778
- [BUGFIX] PromQL: Fix case-insensitive regex label matchers silently dropping matching values. #19167
- [BUGFIX] PromQL: Fix
mad_over_timereturning 0 instead ofNaNwhen the range contains aNaNsample. #19040 - [BUGFIX] Promtool: Accept
--enable-feature=promql-binop-fill-modifiersincheck rules, which previously rejected validfill()/fill_left()/fill_right()expressions. #19153 - [BUGFIX] Remote write: Respect the
AZURE_FEDERATED_TOKEN_FILEenvironment variable for workload identity authentication instead of hardcoding the token file path. #18973 - [BUGFIX] Rules: Clean up stale
rule_group_last_rule_duration_sum_secondsandrule_group_last_restore_duration_secondsseries when a rule group is removed or renamed on reload. Previously each reload leaked two series per dropped group, growing/metricscardinality over time. #19107 - [BUGFIX] Scrape: Fix scrape manager spinning at 100% CPU on shutdown. #19149
- [BUGFIX] TSDB: Fix silent data loss and potential crash loop when
stale_series_compaction_thresholdis used in the config file. #19016 - [BUGFIX] TSDB: Fix potential data loss on restart when out-of-order ingestion is enabled and blocks are compacted. #19016
- [BUGFIX] TSDB: Fix
prometheus_tsdb_head_stale_seriesover-counting and early eviction of series that change between float, integer histogram, and float histogram sample types. #19183 - [BUGFIX] TSDB: Fix goroutine and file handle leaks when Prometheus fails to open a corrupt TSDB. On Windows, the leaked directory handle also prevented TSDB directory removal. #18291
- [BUGFIX] TSDB: Fix out-of-order queries blocking compaction for hours, causing memory usage to grow. #19013
- [BUGFIX] TSDB: Fix deleted series causing missing samples and errors after restart. #19140
- [BUGFIX] TSDB: Fix native histogram data becoming incorrect after restart. #19202
- [BUGFIX] TSDB: Surface query errors that were previously silently discarded. #19120
- [BUGFIX] TSDB: Honour the configured float chunk encoding when compaction rewrites chunks; previously chunks encoded with
--enable-feature=xor2-encodingcould silently revert to XOR after compaction. #19145 - [BUGFIX] UI: Show the delete-series form on the TSDB Status page when
--web.enable-admin-apiis enabled. #19025
3.13.2 / 2026-07-29
What's Changed
- [SECURITY] Bump golang.org/x/text to v0.39.0 (CVE-2026-56852) and google.golang.org/grpc to v1.82.1 (GHSA-hrxh-6v49-42gf). #19290 by @krajorama
- [BUGFIX] PromQL: Preallocate the active query tracker file to avoid SIGBUS crashes when the data disk is full. #19289 by @akshajrawat
Full Changelog: v3.13.1...v3.13.2
3.13.1 / 2026-07-10
This is a bugfix release for 3.13 LTS.
- [BUGFIX] TSDB: Fix the head-chunk cache returning samples from the wrong chunk, or spurious not-found errors, to range queries after head-chunk truncation. #19134
3.5.5 / 2026-07-09
This release is built with Go 1.25.12 and fixes a security issue in a UI dependency.
- [SECURITY] UI: Bump
sanitize-htmlto v2.17.5 to fix CVE-2026-53606. #19060
3.13.0 / 2026-07-01
This is a Long Term Support LTS release.
- [SECURITY] UI: Bump
sanitize-htmlto fix a cross-site scripting vulnerability (CVE-2026-44990). #18697 - [CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at
/assets/third-party-licenses.txt, replacing thenpm_licenses.tar.bz2archive previously shipped in release tarballs and container images. #18997 - [CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. #18927
- [CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). #18949
- [CHANGE] promtool: Relative file paths in the file passed to
--http.config.fileare now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. #18949 - [CHANGE] PromQL: Rename the
min()andmax()duration-expression functions (experimental feature flagexperimental-duration-expr) tomin_of()andmax_of()to avoid confusion with theminandmaxaggregate operators. #18687 - [FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. #18573
- [FEATURE] Discovery/AWS: Add ability to filter RDS instances. #18859
- [FEATURE] PromQL: Add
min_of(a, b)andmax_of(a, b)scalar experimental functions, returning the smaller or larger of two scalar values. #18687 - [FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. #18564
- [FEATURE] PromQL: Expose per-query
samplesRead(andsamplesReadPerStepwithstats=alland thepromql-per-step-statsfeature flag) in the query stats response, and add theprometheus_engine_query_samples_read_totalengine counter.samplesReadreflects storage I/O distinct fromtotalQueryableSamples, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). #18081 - [FEATURE] Scrape: Add
__convert_classic_histograms_to_nhcb__internal label to allow per-target override ofconvert_classic_histograms_to_nhcbscrape configuration via relabeling. #18840 - [FEATURE] TSDB: Add
storage.tsdb.chunk_encoding.floatsconfiguration field to select float chunk encoding (xororxor2) at runtime, independently of the--enable-feature=xor2-encodingflag. #18769 - [FEATURE] remote_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. #18217
- [FEATURE] Scrape: Add
__always_scrape_classic_histograms__and__scrape_native_histograms__internal labels to allow per-target override of thealways_scrape_classic_histogramsandscrape_native_histogramsscrape configuration via relabeling. #18929 - [ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18791
- [ENHANCEMENT] PromQL: Prettify
fill_left(x) fill_right(x)asfill(x)when both fill values are equal. #18851 - [ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. #18894
- [PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to ~2x faster). #18540
- [PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by ~12-15% in benchmarks. #18699
- [PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (
--enable-feature=created-timestamp-zero-ingestion). #18813 - [BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. #18845
- [BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. #18629
- [BUGFIX] PromQL: A range query whose
endwas not aligned tostepcaused subqueries inside it to evaluate past the parent's last actual step, inflatingpeakSamplesin the query stats and against thequery.max-sampleslimit, and wasting storage I/O reading samples that were never used in the result. #18081 - [BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an
@modifier (e.g.predict_linear(metric[60s] @ T, X)) silently under-countedtotalQueryableSamplesfor steps after step 0. #18081 - [BUGFIX] PromQL: Fix
fill_left/fill_rightproducing missing samples in range queries when usinggroup_left/group_right. #18850 - [BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. #18906
- [BUGFIX] PromQL: Fix panic on
1[5m] smoothedand similar expressions when extended range selectors are enabled. #18764 - [BUGFIX] PromQL: Fix panic when a
smoothedinstant vector selector produces no samples for a series. #18943 - [BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g.
foo offset -(5)). #18768 - [BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces
{}. Via prometheus/common v0.69.0. #18949 - [BUGFIX] Promtool: Fix
check healthyandcheck readywhen--urlends with a trailing slash. #18854 - [BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. #18733
- [BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy
private_iporpublic_ipfield, but do have private NICs attached. #18772 - [BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). #18838
- [BUGFIX] UI: Escape label values offered by PromQL autocomplete. #18658
- [BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. #18739
- [BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. #18847
- [BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. #18849
3.13.0-rc.1 / 2026-06-22
Release notes of the 3.13-rc.1 release:
The 3.13.0-rc.0 release was only partially successful due to the migration from NPM to PNPM and subsequent CI issues, so most of the changes in this release candidate are CI/build-related. The only user-facing change is:
- [CHANGE] UI: Third-party npm dependency licenses are now embedded in the Prometheus binary and served at
/assets/third-party-licenses.txt, replacing thenpm_licenses.tar.bz2archive previously shipped in release tarballs and container images. #18997
Release notes of the 3.13-rc.0 release, as it was not published in partial state:
- [SECURITY] UI: Bump
sanitize-htmlto fix a cross-site scripting vulnerability (CVE-2026-44990). #18697 - [CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. #18927
- [CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). #18949
- [CHANGE] promtool: Relative file paths in the file passed to
--http.config.fileare now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. #18949 - [CHANGE] PromQL: Rename the
min()andmax()duration-expression functions (experimental feature flagexperimental-duration-expr) tomin_of()andmax_of()to avoid confusion with theminandmaxaggregate operators. #18687 - [FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. #18573
- [FEATURE] Discovery/AWS: Add ability to filter RDS instances. #18859
- [FEATURE] PromQL: Add
min_of(a, b)andmax_of(a, b)scalar experimental functions, returning the smaller or larger of two scalar values. #18687 - [FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. #18564
- [FEATURE] PromQL: Expose per-query
samplesRead(andsamplesReadPerStepwithstats=alland thepromql-per-step-statsfeature flag) in the query stats response, and add theprometheus_engine_query_samples_read_totalengine counter.samplesReadreflects storage I/O distinct fromtotalQueryableSamples, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). #18081 - [FEATURE] Scrape: Add
__convert_classic_histograms_to_nhcb__internal label to allow per-target override ofconvert_classic_histograms_to_nhcbscrape configuration via relabeling. #18840 - [FEATURE] TSDB: Add
storage.tsdb.chunk_encoding.floatsconfiguration field to select float chunk encoding (xororxor2) at runtime, independently of the--enable-feature=xor2-encodingflag. #18769 - [FEATURE] remote_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. #18217
- [FEATURE] Scrape: Add
__always_scrape_classic_histograms__and__scrape_native_histograms__internal labels to allow per-target override of thealways_scrape_classic_histogramsandscrape_native_histogramsscrape configuration via relabeling. #18929 - [ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18791
- [ENHANCEMENT] PromQL: Prettify
fill_left(x) fill_right(x)asfill(x)when both fill values are equal. #18851 - [ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. #18894
- [PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to ~2x faster). #18540
- [PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by ~12-15% in benchmarks. #18699
- [PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (
--enable-feature=created-timestamp-zero-ingestion). #18813 - [BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. #18845
- [BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. #18629
- [BUGFIX] PromQL: A range query whose
endwas not aligned tostepcaused subqueries inside it to evaluate past the parent's last actual step, inflatingpeakSamplesin the query stats and against thequery.max-sampleslimit, and wasting storage I/O reading samples that were never used in the result. #18081 - [BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an
@modifier (e.g.predict_linear(metric[60s] @ T, X)) silently under-countedtotalQueryableSamplesfor steps after step 0. #18081 - [BUGFIX] PromQL: Fix
fill_left/fill_rightproducing missing samples in range queries when usinggroup_left/group_right. #18850 - [BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. #18906
- [BUGFIX] PromQL: Fix panic on
1[5m] smoothedand similar expressions when extended range selectors are enabled. #18764 - [BUGFIX] PromQL: Fix panic when a
smoothedinstant vector selector produces no samples for a series. #18943 - [BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g.
foo offset -(5)). #18768 - [BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces
{}. Via prometheus/common v0.69.0. #18949 - [BUGFIX] Promtool: Fix
check healthyandcheck readywhen--urlends with a trailing slash. #18854 - [BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. #18733
- [BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy
private_iporpublic_ipfield, but do have private NICs attached. #18772 - [BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). #18838
- [BUGFIX] UI: Escape label values offered by PromQL autocomplete. #18658
- [BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. #18739
- [BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. #18847
- [BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. #18849
3.13.0-rc.0 / 2026-06-18
- [SECURITY] UI: Bump
sanitize-htmlto fix a cross-site scripting vulnerability (CVE-2026-44990). #18697 - [CHANGE] API: Use SHA-256 instead of SHA-1 to generate rule group pagination tokens. #18927
- [CHANGE] HTTP clients: Credentials (Authorization header, basic auth, bearer token, OAuth2, configured headers) are no longer forwarded when following a redirect to a different host; affects scraping, remote read/write, alerting, and service discovery. Via prometheus/common v0.69.0 (CVE-2025-4673 CVE-2023-45289). #18949
- [CHANGE] promtool: Relative file paths in the file passed to
--http.config.fileare now resolved relative to that config file's directory instead of its parent directory. Via prometheus/common v0.69.0. #18949 - [CHANGE] PromQL: Rename the
min()andmax()duration-expression functions (experimental feature flagexperimental-duration-expr) tomin_of()andmax_of()to avoid confusion with theminandmaxaggregate operators. #18687 - [FEATURE] API: Add experimental search endpoints to search metric names, label names, and label values. #18573
- [FEATURE] Discovery/AWS: Add ability to filter RDS instances. #18859
- [FEATURE] PromQL: Add
min_of(a, b)andmax_of(a, b)scalar experimental functions, returning the smaller or larger of two scalar values. #18687 - [FEATURE] PromQL: Add support for smoothed/anchored rate with native histograms. #18564
- [FEATURE] PromQL: Expose per-query
samplesRead(andsamplesReadPerStepwithstats=alland thepromql-per-step-statsfeature flag) in the query stats response, and add theprometheus_engine_query_samples_read_totalengine counter.samplesReadreflects storage I/O distinct fromtotalQueryableSamples, which counts samples loaded into the evaluator (and so over-counts when a sample is reused across multiple range-vector windows). #18081 - [FEATURE] Scrape: Add
__convert_classic_histograms_to_nhcb__internal label to allow per-target override ofconvert_classic_histograms_to_nhcbscrape configuration via relabeling. #18840 - [FEATURE] TSDB: Add
storage.tsdb.chunk_encoding.floatsconfiguration field to select float chunk encoding (xororxor2) at runtime, independently of the--enable-feature=xor2-encodingflag. #18769 - [FEATURE] remote_write: Add Certificate support for ingesting data into an Azure Monitor Workspace. #18217
- [FEATURE] Scrape: Add
__always_scrape_classic_histograms__and__scrape_native_histograms__internal labels to allow per-target override of thealways_scrape_classic_histogramsandscrape_native_histogramsscrape configuration via relabeling. #18929 - [ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18791
- [ENHANCEMENT] PromQL: Prettify
fill_left(x) fill_right(x)asfill(x)when both fill values are equal. #18851 - [ENHANCEMENT] UI: Improve autocompletion after closing a function bracket. #18894
- [PERF] Labels: Add case-insensitive prefix matching to speed up evaluation of long case-insensitive regular expressions (up to ~2x faster). #18540
- [PERF] TSDB: Reduce per-sample overhead in chunk population, speeding up affected queries by ~12-15% in benchmarks. #18699
- [PERF] TSDB: Eliminate unnecessary heap allocations in the V2 histogram WAL decoder, reducing allocations by up to 50% and memory by up to 10% for deployments using native histograms with created-timestamp storage enabled (
--enable-feature=created-timestamp-zero-ingestion). #18813 - [BUGFIX] Discovery/AWS: Fix failure when processing an AWS RDS cluster without instances. #18845
- [BUGFIX] Fix race condition in initTime that could cause ErrOutOfBounds. #18629
- [BUGFIX] PromQL: A range query whose
endwas not aligned tostepcaused subqueries inside it to evaluate past the parent's last actual step, inflatingpeakSamplesin the query stats and against thequery.max-sampleslimit, and wasting storage I/O reading samples that were never used in the result. #18081 - [BUGFIX] PromQL: A range query containing an at-modifier-unsafe function over a range-vector with an
@modifier (e.g.predict_linear(metric[60s] @ T, X)) silently under-countedtotalQueryableSamplesfor steps after step 0. #18081 - [BUGFIX] PromQL: Fix
fill_left/fill_rightproducing missing samples in range queries when usinggroup_left/group_right. #18850 - [BUGFIX] PromQL: Fix for resets() and changes() in anchored range extenders with histograms. #18906
- [BUGFIX] PromQL: Fix panic on
1[5m] smoothedand similar expressions when extended range selectors are enabled. #18764 - [BUGFIX] PromQL: Fix panic when a
smoothedinstant vector selector produces no samples for a series. #18943 - [BUGFIX] PromQL: Fix panic when using a parenthesised plain number as an offset (e.g.
foo offset -(5)). #18768 - [BUGFIX] promtool: Fix panic when parsing exposition text containing empty braces
{}. Via prometheus/common v0.69.0. #18949 - [BUGFIX] Promtool: Fix
check healthyandcheck readywhen--urlends with a trailing slash. #18854 - [BUGFIX] Rules: Close PromQL query after each rule evaluation to ensure resources are released. #18733
- [BUGFIX] Scaleway SD: Resolve VPC/IPAM-only instances that have no legacy
private_iporpublic_ipfield, but do have private NICs attached. #18772 - [BUGFIX] TSDB: Do not leak head series when an integer histogram append is rejected (e.g. out-of-order). #18838
- [BUGFIX] UI: Escape label values offered by PromQL autocomplete. #18658
- [BUGFIX] TSDB: Fix chunk snapshot encoding for EncXOR2 chunks, preventing corruption on TSDB restart when EncXOR2-encoded series were present. #18739
- [BUGFIX] TSDB: Store a millisecond timestamp (not a WAL segment number) in walExpiries when a series is evicted via CompactStaleHead/CompactSelectedSeries, so the series's label record is correctly retained in the next WAL checkpoint and replays cleanly. #18847
- [BUGFIX] TSDB: Prevent loss of samples at the chunk-range boundary when CompactSelectedSeries (and CompactStaleHead) evict the series — the per-slice compaction loop now runs one more iteration so the boundary timestamp is captured in a block before the in-memory copy is removed. #18849
3.5.4 / 2026-06-17
This release fixes multiple security issues.
- [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via
/-/configendpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18650 - [SECURITY] Dependencies: Bump
golang.org/x/netto v0.55.0 and OpenTelemetry to v1.43.0 to fix reported CVEs (GO-2026-5026, GO-2026-4918, GO-2026-4985). #18934 - [SECURITY] UI: Bump mantine-ui dependencies (
react-router-dom,vitest,vite,postcss) to their patched versions to resolve security advisories. #18935 - [ENHANCEMENT] Release: Container images are now also published to the GitHub Container Registry (ghcr.io). #18792
3.12.0 / 2026-05-28
This release contains security fixes, new features (especially around PromQL and Service Discovery), performance improvements in TSDB, Start Timestamp improvements and numerous bug fixes.
Thanks to all contributors!
Key Highlights
- Security: Two security vulnerabilities have been addressed: a denial of service in remote-write (snappy decompression limit) and a secret exposure leak in STACKIT service discovery.
- PromQL & Metadata: Several features and bug fixes related to the experimental "start timestamps" support, including updates to
rate(),irate(),increase(), andresets(). New experimental functionsstart(),end(),range(), andstep()are introduced. - TSDB Performance: Optimizations in head chunk lookup (constant time) and mmap operations to reduce CPU usage.
- Service Discovery: Added support for DigitalOcean Managed Databases and Outscale VM, along with improvements to AWS SD (IPv6 support for EC2, external ID support).
- UI: Added a web interface for deleting time series and cleaning tombstones.
Changelog
- [SECURITY] Remote: Reject snappy-compressed received requests via Remote Write whose declared decoded length exceeds the 32MB. Thanks to @hibrian827 for reporting it. #18642
- [SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via
/-/configendpoint. Thanks to @August829 and @Phaxma for reporting. GHSA-39j6-789q-qxvh #18649 - [CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used
st-storageflag is enabled. #18221 - [FEATURE] API: Add
/api/v1/status/self_metricsendpoint returning the current state of the Prometheus server's own metrics about itself as JSON. #18411 - [FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery #18287
- [FEATURE] Prometheus: Add support for the aix/ppc64 compilation target #18321
- [FEATURE] Discovery: Add Outscale VM service discovery (
outscale_sd_configs) for discovering scrape targets from the Outscale Cloud API. #18139 - [FEATURE] PromQL: Emit a warning when
sort,sort_by_labelorsort_by_label_descis used within range (matrix) queries, as these functions do not have effect in that context. #18498 - [FEATURE] PromQL: Add
start(),end(),range(), andstep()experimental functions #17877 - [FEATURE] PromQL: Update
resets()function to consider start timestamp resets. Hidden behinduse-start-timestampsfeature flag. #18627 - [FEATURE] Prometheus: Promote auto-reload-config as stable #18620
- [FEATURE] TSDB/Agent: Add
CheckpointFromInMemorySeriesoption toagent.DBthat enables checkpoint based on in-memory series. #17948 - [FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. #18390
- [FEATURE] PromQL: Use start timestamps for
rate(),irate(), andincrease()calculations, behind a feature flaguse-start-timestamps. Doesn't work together with extended range selectorsanchoredandsmoothed. #18344 - [FEATURE] Scrape: Added a feature flag
st-synthesiswhich synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. #18279 - [FEATURE] promqltest: support
@stannotation inloadblocks to specify per-sample start timestamps. #18360 - [ENHANCEMENT] API: reject concurrent fgprof profiles. #18651
- [ENHANCEMENT] AWS SD: Add optional
external_idfield to ECS/MSK/RDS/Elasticache. #18579 - [ENHANCEMENT] AWS SD: Add optional
external_idfield. #17171 - [ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. #18187
- [ENHANCEMENT] Promtool: Add
--headerflag toquery instantcommand, matching existingquery rangebehaviour. #18418 - [ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. #16088
- [PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time #18302
- [PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. #18541
- [PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count #18272
- [PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. #18568
- [BUGFIX] PromQL: Fix
info()function incorrectly handling negated__name__matchers #17932 - [BUGFIX] API: Return duration expressions in
/parse_ast. #18624 - [BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec #18305
- [BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 #18567
- [BUGFIX] Config: Validate remote_write queue_config fields at load time to prevent runtime panic and silent misconfiguration. #18209
- [BUGFIX] Discovery/Consul: Add
health_filterfor Health API filtering, fixing breakage when using Catalog-only fields likeServiceTagsinfilter. #18479 #18499 - [BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. #18408
- [BUGFIX] PromQL: Fix
smoothedrate/increase returning zero instead of no result when all data falls strictly after the query range. #18523 - [BUGFIX] PromQL: Fix metric name not being dropped when last_over_time or first_over_time is applied to subqueries containing name-dropping functions like abs(). #18409
- [BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. #18660
- [BUGFIX] PromQL: Fix parsing of
range()keyword in duration expressions such asfoo[5m+range()]. #18623 - [BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the
@modifier is used. #18531 - [BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. #18639
- [BUGFIX] Scrape: Fix panic when scraping malformed native histograms. #18414
- [BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. #18382
- [BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. #18421
- [BUGFIX] TSDB: Allow retention percentage with new data path. #18628
- [BUGFIX] TSDB: Preserve decimal precision in percentage-based retention #18374
- [BUGFIX] TSDB: fix prometheus_tsdb_head_chunks going negative after WAL replay #18401
- [BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. #18692
- [BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing #18469
- [BUGFIX] UI: Escape label values offered by PromQL autocomplete. #18658
- [BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. #18682
- [BUGFIX]
prometheus_sd_refresh*andprometheus_sd_discovered_targetsmetrics for specific scrape jobs are deleted when the scrape job is removed. #17614 - [BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. #18641
- [BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. #18292
- [BUGFIX] Config: Update
--enable-featureflag description and sort feature names. #18487