Skip to content

chore: add additional metric to show time since last block #27

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
auricom merged 1 commit into from
Dec 19, 2025
Merged

chore: add additional metric to show time since last block #27

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,56 @@ When metrics are enabled, the following metrics are exposed:
- **Labels**: `chain_id`, `type`
- **Description**: Latest DA height for header and data submissions

### Block Time Metrics

### `ev_metrics_block_time_seconds`
- **Type**: Histogram
- **Labels**: `chain_id`
- **Description**: Time between consecutive blocks with histogram buckets for accurate SLO calculations
- **Buckets**: 0.1, 0.15, 0.2, 0.25, 0.3, 0.35, 0.4, 0.45, 0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95, 1, 1.5, 2 seconds

### `ev_metrics_block_time_summary_seconds`
- **Type**: Summary
- **Labels**: `chain_id`
- **Description**: Block time with percentiles over a 60-second rolling window
- **Note**: Will show NaN when no blocks have been received in the last 60 seconds

### `ev_metrics_time_since_last_block_seconds`
- **Type**: Gauge
- **Labels**: `chain_id`
- **Description**: Seconds since last block was received. Use this metric for alerting on stale blocks.
- **Alerting**: Alert when this value exceeds 60 seconds to detect block production issues before summary metrics show NaN

### `ev_metrics_block_time_slo_seconds`
- **Type**: Gauge
- **Labels**: `chain_id`, `quantile`
- **Description**: SLO thresholds for block time
- **Values**:
- `0.5`: 2.0s
- `0.9`: 3.0s
- `0.95`: 4.0s
- `0.99`: 5.0s

### `ev_metrics_block_receive_delay_seconds`
- **Type**: Histogram
- **Labels**: `chain_id`
- **Description**: Delay between block creation and reception with histogram buckets
- **Buckets**: 0.1, 0.25, 0.5, 1.0, 2.0, 3.0, 5.0, 10.0, 15.0, 30.0, 60.0 seconds

### `ev_metrics_block_receive_delay_slo_seconds`
- **Type**: Gauge
- **Labels**: `chain_id`, `quantile`
- **Description**: SLO thresholds for block receive delay
- **Values**:
- `0.5`: 1.0s
- `0.9`: 3.0s
- `0.95`: 5.0s
- `0.99`: 10.0s
Comment on lines +153 to +197
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The new section headers like Block Time Metrics use the same heading level (###) as the individual metric names that follow (e.g., ### ev_metrics_block_time_seconds``). This creates a flat and somewhat confusing document structure. For better hierarchy and readability, consider demoting the individual metric headings to a lower level (####).

For example:

### Block Time Metrics

#### `ev_metrics_block_time_seconds`
- **Type**: Histogram
...

#### `ev_metrics_block_time_summary_seconds`
- **Type**: Summary
...

This would apply to all new metrics documented under the Block Time Metrics group.


### JSON-RPC Monitoring Metrics

When `--evm-rpc-url` is provided:

### `ev_metrics_jsonrpc_request_duration_seconds`
- **Type**: Histogram
- **Labels**: `chain_id`
Expand Down
2 changes: 2 additions & 0 deletions pkg/exporters/verifier/verifier.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,8 @@ func (e *exporter) ExportMetrics(ctx context.Context, m *metrics.Metrics) error
case <-refreshTicker.C:
// ensure that submission duration is always included in the 60 second window.
m.RefreshSubmissionDuration()
// update time since last block metric
m.UpdateTimeSinceLastBlock()
case header := <-headers:
// record block arrival time for millisecond precision
arrivalTime := time.Now()
Expand Down
25 changes: 25 additions & 0 deletions pkg/metrics/metrics.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,8 @@ type Metrics struct {
BlockTime *prometheus.HistogramVec
// BlockTimeSummary tracks block time with percentiles over a rolling window.
BlockTimeSummary *prometheus.SummaryVec
// TimeSinceLastBlock tracks seconds since last block was received.
TimeSinceLastBlock *prometheus.GaugeVec
// BlockReceiveDelay tracks the delay between block creation and reception with histogram buckets.
BlockReceiveDelay *prometheus.HistogramVec
// JsonRpcRequestDuration tracks the duration of JSON-RPC requests to the EVM node.
Expand Down Expand Up @@ -181,6 +183,14 @@ func NewWithRegistry(namespace string, registerer prometheus.Registerer) *Metric
},
[]string{"chain_id"},
),
TimeSinceLastBlock: factory.NewGaugeVec(
prometheus.GaugeOpts{
Namespace: namespace,
Name: "time_since_last_block_seconds",
Help: "seconds since last block was received",
},
[]string{"chain_id"},
),
BlockReceiveDelay: factory.NewHistogramVec(
prometheus.HistogramOpts{
Namespace: namespace,
Expand Down Expand Up @@ -554,6 +564,21 @@ func (m *Metrics) RecordBlockTime(chainID string, arrivalTime time.Time) {

// update last seen arrival time
m.lastBlockArrivalTime[chainID] = arrivalTime
// reset time since last block to 0
m.TimeSinceLastBlock.WithLabelValues(chainID).Set(0)
Comment on lines 566 to +568
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot Dec 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

There's a potential issue here with out-of-order block arrivals (based on arrivalTime). If a block arrives with a timestamp earlier than the lastArrival, lastBlockArrivalTime is updated with this older time, and TimeSinceLastBlock is reset to 0. The next time UpdateTimeSinceLastBlock runs, it will calculate a large timeSince value based on the now-older lastArrival, leading to a sudden incorrect spike in the time_since_last_block_seconds metric. Both lastBlockArrivalTime and TimeSinceLastBlock should only be updated for blocks that arrive in order (i.e., with a later arrivalTime).

A full refactoring of the RecordBlockTime function would be best to address this robustly:

func (m *Metrics) RecordBlockTime(chainID string, arrivalTime time.Time) {
	m.mu.Lock()
	defer m.mu.Unlock()

	lastArrival, exists := m.lastBlockArrivalTime[chainID]
	if exists {
		if !arrivalTime.After(lastArrival) {
			// Ignore out-of-order or same-time arrival to prevent incorrect metrics.
			return
		}
		blockTime := arrivalTime.Sub(lastArrival)
		m.BlockTime.WithLabelValues(chainID).Observe(blockTime.Seconds())
		m.BlockTimeSummary.WithLabelValues(chainID).Observe(blockTime.Seconds())
	}

	// update last seen arrival time
	m.lastBlockArrivalTime[chainID] = arrivalTime
	// reset time since last block to 0
	m.TimeSinceLastBlock.WithLabelValues(chainID).Set(0)
}

}

// UpdateTimeSinceLastBlock updates the time_since_last_block metric for all chains
// should be called periodically to keep the metric current.
func (m *Metrics) UpdateTimeSinceLastBlock() {
m.mu.Lock()
defer m.mu.Unlock()

now := time.Now()
for chainID, lastArrival := range m.lastBlockArrivalTime {
timeSince := now.Sub(lastArrival).Seconds()
m.TimeSinceLastBlock.WithLabelValues(chainID).Set(timeSince)
}
}

// RecordBlockReceiveDelay records the delay between block creation and reception
Expand Down