Skip to content

PromQL: use start timestamps for rate()-like calculations#18344

Merged
krajorama merged 21 commits into
prometheus:mainfrom
vpranckaitis:use_start_timestamps_in_rate_like_functions
Apr 21, 2026
Merged

PromQL: use start timestamps for rate()-like calculations#18344
krajorama merged 21 commits into
prometheus:mainfrom
vpranckaitis:use_start_timestamps_in_rate_like_functions

Conversation

@vpranckaitis
Copy link
Copy Markdown
Contributor

@vpranckaitis vpranckaitis commented Mar 23, 2026

Which issue(s) does the PR fix:

Implemented a change to use start timestamps for rate() and increase() calculations. This is a part of OTLP Delta Support project.

After this change, counter resets are not only detected by looking at datapoint values, but also by checking start timestamps. As described in PROM-77 proposal, this allows querying deltas with rate() and increase() functions, as long as they have valid start timestamps. Additionally, it should also improve counter reset detection for cumulative counters, where the first scraped value after a counter reset is as high as it was before the reset.

This PR also follows some of the ideas expressed in this comment by @enisoc which are meant to minimize memory usage impact when start timestamps usage is not enabled or when their values would be not useful for the PromQL evaluation.

This PR does not include the changes necessary for start timestamps to work when anchored and smoothed modifiers are used. It was agreed in OTLP Delta Support Working Group meetings that we want to first see a working end-to-end solution before addressing experimental anchored and smoothed modifiers.

Does this PR introduce a user-facing change?

[FEATURE] Use start timestamps for `rate()`, `irate()`, and `increase()` calculations, behind a feature flag `use-start-timestamps`. Doesn't work together with extended range selectors `anchored` and `smoothed`.

Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>
Comment thread promql/engine_test.go Outdated
Copy link
Copy Markdown
Member

@krajorama krajorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some minor comments, this is looking pretty good

Comment thread tsdb/db.go
Comment thread promql/engine.go
Comment thread promql/functions.go
Comment thread promql/functions.go Outdated
Comment thread promql/functions.go Outdated
Comment thread docs/feature_flags.md Outdated
Comment thread promql/functions.go Outdated
Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>
Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>
@vpranckaitis vpranckaitis requested a review from krajorama March 24, 2026 12:45
Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>
@vpranckaitis vpranckaitis changed the title PromQL: use start timestamps for rate() and increase() calculations PromQL: use start timestamps for rate()-like calculations Mar 31, 2026
@vpranckaitis vpranckaitis force-pushed the use_start_timestamps_in_rate_like_functions branch from 28f1519 to b5682a1 Compare April 3, 2026 07:26
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
@vpranckaitis vpranckaitis marked this pull request as ready for review April 3, 2026 08:39
@vpranckaitis vpranckaitis requested review from a team and roidelapluie as code owners April 3, 2026 08:39
Copy link
Copy Markdown
Member

@krajorama krajorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, almost there. I've tried a number of times to wrap my head around the T=ST case ... seems ok, but maybe we could walk through it at the next WG meeting.
I've made a bunch of suggestions to the tests and code.

Comment thread promql/engine.go Outdated
Comment thread promql/functions.go Outdated
Comment thread promql/functions.go Outdated
Comment thread promql/functions.go Outdated
Comment thread promql/promqltest/testdata/start_timestamps.test
Comment thread promql/promqltest/testdata/start_timestamps.test Outdated
Comment thread promql/promqltest/testdata/start_timestamps.test
Comment thread promql/functions.go
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
@vpranckaitis vpranckaitis requested a review from krajorama April 13, 2026 10:58
Copy link
Copy Markdown
Member

@krajorama krajorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Last 3 comments.

Comment thread promql/functions.go Outdated
Comment thread promql/functions.go Outdated
Comment thread docs/feature_flags.md Outdated
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
@vpranckaitis vpranckaitis requested a review from krajorama April 14, 2026 09:45
Copy link
Copy Markdown
Member

@krajorama krajorama left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.
I think it's too early to add it to features API. We'll need native histograms support for that and extended range selectors.

Comment thread promql/functions.go Outdated
vamsi-01 added a commit to vamsi-01/prometheus that referenced this pull request Apr 20, 2026
Implement detection of overlapping delta aggregation windows where
a sample's start timestamp is less than the previous sample's timestamp.

This resolves issue prometheus#18534 by:
- Collecting start timestamps during matrix evaluation
- Checking for overlaps in both float and histogram samples
- Emitting DeltaStartTimeOverlapWarning annotations when detected
- Warning once per series to avoid spam

The implementation builds on PR prometheus#18344's start timestamp infrastructure,
which propagates ST values through the query path via StartTimestamps struct.

Changes:
- Add DeltaStartTimeOverlapWarning annotation type with merging support
- Implement checkDeltaStartTimeOverlaps() to detect overlapping windows
- Enable ST collection in matrixSelector for overlap detection
- Add comprehensive unit tests for overlap detection logic
- Test various scenarios: normal case, overlaps, nil/zero STs

Depends on: prometheus#18344 (PromQL: use start timestamps for rate-like calculations)
Fixes: prometheus#18534

```release-notes
[FEATURE] PromQL: Annotate a warning when delta samples have overlapping start times (start time < previous timestamp).
```

Signed-off-by: Vamsi Mathala <vmathala@redhat.com>
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
@krajorama krajorama enabled auto-merge (squash) April 21, 2026 14:48
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
auto-merge was automatically disabled April 21, 2026 14:53

Head branch was pushed to by a user without write access

@krajorama krajorama merged commit 321fe34 into prometheus:main Apr 21, 2026
34 checks passed
vamsi-01 added a commit to vamsi-01/prometheus that referenced this pull request Apr 21, 2026
Implement detection of overlapping delta aggregation windows where
a sample's start timestamp is less than the previous sample's timestamp.

This resolves issue prometheus#18534 by:
- Collecting start timestamps during matrix evaluation
- Checking for overlaps in both float and histogram samples
- Emitting DeltaStartTimeOverlapWarning annotations when detected
- Warning once per series to avoid spam

The implementation builds on PR prometheus#18344's start timestamp infrastructure,
which propagates ST values through the query path via StartTimestamps struct.

Changes:
- Add DeltaStartTimeOverlapWarning annotation type with merging support
- Implement checkDeltaStartTimeOverlaps() to detect overlapping windows
- Enable ST collection in matrixSelector for overlap detection
- Add comprehensive unit tests for overlap detection logic
- Test various scenarios: normal case, overlaps, nil/zero STs

Depends on: prometheus#18344 (PromQL: use start timestamps for rate-like calculations)
Fixes: prometheus#18534

```release-notes
[FEATURE] PromQL: Annotate a warning when delta samples have overlapping start times (start time < previous timestamp).
```

Signed-off-by: Vamsi Mathala <vmathala@redhat.com>
vamsi-01 added a commit to vamsi-01/prometheus that referenced this pull request Apr 21, 2026
Implement detection of overlapping delta aggregation windows where
a sample's start timestamp is less than the previous sample's timestamp.

This resolves issue prometheus#18534 by:
- Collecting start timestamps during matrix evaluation
- Checking for overlaps in both float and histogram samples
- Emitting DeltaStartTimeOverlapWarning annotations when detected
- Warning once per series to avoid spam

The implementation builds on PR prometheus#18344's start timestamp infrastructure,
which propagates ST values through the query path via StartTimestamps struct.

Changes:
- Add DeltaStartTimeOverlapWarning annotation type with merging support
- Implement checkDeltaStartTimeOverlaps() to detect overlapping windows
- Enable ST collection in matrixSelector for overlap detection
- Add comprehensive unit tests for overlap detection logic
- Test various scenarios: normal case, overlaps, nil/zero STs

Fixes: prometheus#18534

```release-notes
[FEATURE] PromQL: Annotate a warning when delta samples have overlapping start times (start time < previous timestamp).
```

Signed-off-by: Vamsi Mathala <vmathala@redhat.com>
vamsi-01 added a commit to vamsi-01/prometheus that referenced this pull request Apr 22, 2026
Implement detection of overlapping aggregation windows where a sample's
start timestamp is less than the previous sample's timestamp. This applies
to both delta and cumulative counter metrics.

This resolves issue prometheus#18534 by:
- Checking for overlaps in rate/increase/delta function calls
- Using condition: currST != 0 && currST < prevT && currST != prevST
- This correctly handles both delta and cumulative counter metrics
- Emitting StartTimeOverlapWarning annotations when detected
- Warning once per series to avoid spam
- Only extracting metric name when overlap is detected (performance)

The implementation builds on PR prometheus#18344's start timestamp infrastructure,
which propagates ST values through the query path via StartTimestamps struct.

Changes:
- Add StartTimeOverlapWarning annotation type with merging support
- Implement checkStartTimeOverlap() to detect overlapping windows
- Add overlap checks in extrapolatedRate() for float samples
- Add overlap checks in histogramRate() for histogram samples
- Add promqltest tests for overlap detection scenarios
- Remove engine.go matrixSelector check (moved to function call path)

Fixes: prometheus#18534
Signed-off-by: Vamsi Mathala <vmathala@redhat.com>
vamsi-01 added a commit to vamsi-01/prometheus that referenced this pull request Apr 23, 2026
Implement detection of overlapping aggregation windows where a sample's
start timestamp is less than the previous sample's timestamp. This applies
to both delta and cumulative counter metrics.

This resolves issue prometheus#18534 by:
- Checking for overlaps in rate/increase/delta function calls
- Using condition: currST != 0 && currST < prevT && currST != prevST
- This correctly handles both delta and cumulative counter metrics
- Emitting StartTimeOverlapWarning annotations when detected
- Warning once per series to avoid spam
- Only extracting metric name when overlap is detected (performance)

The implementation builds on PR prometheus#18344's start timestamp infrastructure,
which propagates ST values through the query path via StartTimestamps struct.

Changes:
- Add StartTimeOverlapWarning annotation type with merging support
- Implement checkStartTimeOverlap() to detect overlapping windows
- Add overlap checks in extrapolatedRate() for float samples
- Add overlap checks in histogramRate() for histogram samples
- Add promqltest tests for overlap detection scenarios
- Remove engine.go matrixSelector check (moved to function call path)

Fixes: prometheus#18534
Signed-off-by: Vamsi Mathala <vmathala@redhat.com>
rbizos pushed a commit to rbizos/prometheus that referenced this pull request Apr 29, 2026
…us#18344)

* PromQL: use start timestamps for rate() and increase() calculations

* implement start timestamps reset detection for `irate()`
* add `start_timestamps.test`
* add a couple of tests with subqueries
* add a test for cumulative with unknown start timestamp
* update `enable-features` CLI parameter description
* `make cli-documentation`

Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>

---------

Signed-off-by: Vilius Pranckaitis <vpranckaitis@gmail.com>
Signed-off-by: vpranckaitis <vpranckaitis@gmail.com>
Signed-off-by: Raphael Bizos <r.bizos@criteo.com>
eleboucher pushed a commit to eleboucher/homelab that referenced this pull request May 28, 2026
…➔ v3.12.0) (#730)

This PR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [quay.io/prometheus/prometheus](https://github.com/prometheus/prometheus) | minor | `v3.11.3` → `v3.12.0` |

---

### Release Notes

<details>
<summary>prometheus/prometheus (quay.io/prometheus/prometheus)</summary>

### [`v3.12.0`](https://github.com/prometheus/prometheus/releases/tag/v3.12.0): 3.12.0 / 2026-05-28

[Compare Source](prometheus/prometheus@v3.11.3...v3.12.0)

- \[SECURITY] Remote-write: Reject snappy-compressed requests whose declared decoded length exceeds the 32MB. Thanks to [@&#8203;hibrian827](https://github.com/hibrian827) for reporting it. [#&#8203;18642](prometheus/prometheus#18642)
- \[SECURITY] STACKIT SD: Fix secrets being exposed in plaintext via `/-/config` endpoint. Thanks to [@&#8203;August829](https://github.com/August829) and [@&#8203;Phaxma](https://github.com/Phaxma) for reporting. GHSA-39j6-789q-qxvh [#&#8203;18649](prometheus/prometheus#18649)
- \[CHANGE] TSDB/Agent: Adds Start Timestamp field to all WAL Histogram samples in memory; used `st-storage` flag is enabled. [#&#8203;18221](prometheus/prometheus#18221)
- \[FEATURE] API: Add `/api/v1/status/self_metrics` endpoint returning the current state of the Prometheus server's own metrics about itself as JSON. [#&#8203;18411](prometheus/prometheus#18411)
- \[FEATURE] Discovery: Add DigitalOcean Managed Databases service discovery [#&#8203;18287](prometheus/prometheus#18287)
- \[FEATURE] Prometheus: Add support for the aix/ppc64 compilation target [#&#8203;18321](prometheus/prometheus#18321)
- \[FEATURE] Discovery: Add Outscale VM service discovery (`outscale_sd_configs`) for discovering scrape targets from the Outscale Cloud API. [#&#8203;18139](prometheus/prometheus#18139)
- \[FEATURE] PromQL: Emit a warning when `sort`, `sort_by_label` or `sort_by_label_desc` is used within range (matrix) queries, as these functions do not have effect in that context. [#&#8203;18498](prometheus/prometheus#18498)
- \[FEATURE] PromQL: Add `start()`, `end()`, `range()`, and `step()` experimental functions [#&#8203;17877](prometheus/prometheus#17877)
- \[FEATURE] PromQL: Update `resets()` function to consider start timestamp resets. Hidden behind `use-start-timestamps` feature flag. [#&#8203;18627](prometheus/prometheus#18627)
- \[FEATURE] Prometheus: Promote auto-reload-config as stable [#&#8203;18620](prometheus/prometheus#18620)
- \[FEATURE] TSDB/Agent: Add `CheckpointFromInMemorySeries` option to `agent.DB` that enables checkpoint based on in-memory series. [#&#8203;17948](prometheus/prometheus#17948)
- \[FEATURE] UI: Add a web interface for deleting time series and cleaning tombstones, accessible from the Status menu. [#&#8203;18390](prometheus/prometheus#18390)
- \[FEATURE] PromQL: Use start timestamps for `rate()`, `irate(), and `increase()`calculations, behind a feature flag`use-start-timestamps`. Doesn't work together with extended range selectors `anchored`and`smoothed\`. [#&#8203;18344](prometheus/prometheus#18344)
- \[FEATURE] Scrape: Added a feature flag `st-synthesis` which synthesizes unknown STs for scraped cumulative metrics. Useful when Remote Writing 2.0 with delta or Otel-based backends. [#&#8203;18279](prometheus/prometheus#18279)
- \[FEATURE] promqltest: support `@st` annotation in `load` blocks to specify per-sample start timestamps. [#&#8203;18360](prometheus/prometheus#18360)
- \[ENHANCEMENT] API: reject concurrent fgprof profiles. [#&#8203;18651](prometheus/prometheus#18651)
- \[ENHANCEMENT] AWS SD: Add optional `external_id` field to ECS/MSK/RDS/Elasticache. [#&#8203;18579](prometheus/prometheus#18579)
- \[ENHANCEMENT] AWS SD: Add optional `external_id` field. [#&#8203;17171](prometheus/prometheus#17171)
- \[ENHANCEMENT] Discovery: Propagate SD target updates faster by introducing dynamic backoff interval instead of static 5s interval for throttling. [#&#8203;18187](prometheus/prometheus#18187)
- \[ENHANCEMENT] Promtool: Add `--header` flag to `query instant` command, matching existing `query range` behaviour. [#&#8203;18418](prometheus/prometheus#18418)
- \[ENHANCEMENT]: AWS SD: Allows EC2 service discovery to discover IPv6 addresses to communicate with target endpoints. The private IPv4 address remains the default when both IPv4 and IPv6 addresses are present. [#&#8203;16088](prometheus/prometheus#16088)
- \[PERF] TSDB: Make head chunk lookup in range queries constant time instead of quadratic time [#&#8203;18302](prometheus/prometheus#18302)
- \[PERF] TSDB: Skip entire stripes in mmapHeadChunks when no series need mmapping, reducing CPU utilization significantly at production-relevant scales. [#&#8203;18541](prometheus/prometheus#18541)
- \[PERF] TSDB: Skip clean series during periodic head chunk mmap using cached head chunk count [#&#8203;18272](prometheus/prometheus#18272)
- \[PERF] PromQL: Address FloatHistogram.KahanAdd performance regression on Go 1.26. [#&#8203;18568](prometheus/prometheus#18568)
- \[BUGFIX] PromQL: Fix `info()` function incorrectly handling negated `__name__` matchers [#&#8203;17932](prometheus/prometheus#17932)
- \[BUGFIX] API: Return duration expressions in `/parse_ast`. [#&#8203;18624](prometheus/prometheus#18624)
- \[BUGFIX] API: correctly document formats accepted for duration query request parameters (step, timeout and lookback delta) in OpenAPI spec [#&#8203;18305](prometheus/prometheus#18305)
- \[BUGFIX] Scrape: AppenderV2 now tracks staleness even when OOO/duplicate series errors happen similar to AppenderV1 [#&#8203;18567](prometheus/prometheus#18567)
- \[BUGFIX] Config: Validate remote\_write queue\_config fields at load time to prevent runtime panic and silent misconfiguration. [#&#8203;18209](prometheus/prometheus#18209)
- \[BUGFIX] Discovery/Consul: Add `health_filter` for Health API filtering, fixing breakage when using Catalog-only fields like `ServiceTags` in `filter`. [#&#8203;18479](prometheus/prometheus#18479) [#&#8203;18499](prometheus/prometheus#18499)
- \[BUGFIX] OTLP: limit decompressed body size for gzip-encoded OTLP write requests. [#&#8203;18408](prometheus/prometheus#18408)
- \[BUGFIX] PromQL: Fix `smoothed` rate/increase returning zero instead of no result when all data falls strictly after the query range. [#&#8203;18523](prometheus/prometheus#18523)
- \[BUGFIX] PromQL: Fix metric name not being dropped when last\_over\_time or first\_over\_time is applied to subqueries containing name-dropping functions like abs(). [#&#8203;18409](prometheus/prometheus#18409)
- \[BUGFIX] PromQL: Fix missing warning when mixing exponential and custom-bucket histograms in stats queries. [#&#8203;18660](prometheus/prometheus#18660)
- \[BUGFIX] PromQL: Fix parsing of `range()` keyword in duration expressions such as `foo[5m+range()]`. [#&#8203;18623](prometheus/prometheus#18623)
- \[BUGFIX] PromQL: Fix smoothed vector selector returning no results in binary operations when the `@` modifier is used. [#&#8203;18531](prometheus/prometheus#18531)
- \[BUGFIX] PromQL: Reject NaN, infinite, and out-of-range duration expressions instead of silently producing an out-of-range time.Duration. [#&#8203;18639](prometheus/prometheus#18639)
- \[BUGFIX] Scrape: Fix panic when scraping malformed native histograms. [#&#8203;18414](prometheus/prometheus#18414)
- \[BUGFIX] Scrape: fix panic when scraping a target exposing a summary with no quantiles via the protobuf format. [#&#8203;18382](prometheus/prometheus#18382)
- \[BUGFIX] Scrape: fix scrape failure log file occasionally not applied after a configuration reload. [#&#8203;18421](prometheus/prometheus#18421)
- \[BUGFIX] TSDB: Allow retention percentage with new data path. [#&#8203;18628](prometheus/prometheus#18628)
- \[BUGFIX] TSDB: Preserve decimal precision in percentage-based retention [#&#8203;18374](prometheus/prometheus#18374)
- \[BUGFIX] TSDB: fix prometheus\_tsdb\_head\_chunks going negative after WAL replay [#&#8203;18401](prometheus/prometheus#18401)
- \[BUGFIX] TSDB: panic with native histograms during query of overlapping chunks. [#&#8203;18692](prometheus/prometheus#18692)
- \[BUGFIX] Tracing: fix startup failure for insecure OTLP HTTP tracing [#&#8203;18469](prometheus/prometheus#18469)
- \[BUGFIX] UI: Escape label values offered by PromQL autocomplete. [#&#8203;18658](prometheus/prometheus#18658)
- \[BUGFIX] UI: Improve Y-axis tick label precision for graph values over small ranges. [#&#8203;18682](prometheus/prometheus#18682)
- \[BUGFIX] `prometheus_sd_refresh*` and `prometheus_sd_discovered_targets` metrics for specific scrape jobs are deleted when the scrape job is removed. [#&#8203;17614](prometheus/prometheus#17614)
- \[BUGFIX] Remote: fixed validation for received RW2 requests when parsing metadata unit symbols. This fixes a case when request would cause (recovered) handler panic. [#&#8203;18641](prometheus/prometheus#18641)
- \[BUGFIX] TSDB/Agent: fix race in agent appender where concurrent appends for the same label set could produce duplicate in-memory series and duplicate WAL records. [#&#8203;18292](prometheus/prometheus#18292)
- \[BUGFIX] Config: Update `--enable-feature`  flag description and sort feature names. [#&#8203;18487](prometheus/prometheus#18487)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about these updates again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My4xMDEuMSIsInVwZGF0ZWRJblZlciI6IjQzLjEwMS4xIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJyZW5vdmF0ZS9jb250YWluZXIiLCJ0eXBlL21pbm9yIl19-->

Reviewed-on: https://git.erwanleboucher.dev/eleboucher/homelab/pulls/730
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants