[Telemetry Data] May 13 Arbitrum Sequencer Stall: eBPF Kernel Reality vs Public RPC (47s Delta)

[kant19801201behax5]

Observation

On May 13, 2026 between 05:15:05 and 05:15:52 UTC, my kernel-level monitoring sensor recorded a severe latency spike to the Arbitrum sequencer.

Sharing this because it may be relevant to relay operators and SUAVE Kettle routing design — specifically the problem of public RPC health signals lagging real sequencer state.

---

Raw Telemetry

Measurement method: Linux kernel tcp_connect kprobe, 150ms probe interval, target sequencer.arbitrum.io:443.

[
  {"ts": 1778649305, "chain": "arbitrum", "rtt_ms": 14,   "risk": 0.05, "stall_flag": false},
  {"ts": 1778649306, "chain": "arbitrum", "rtt_ms": 3081, "risk": 0.97, "stall_flag": true,  "note": "STALL ONSET"},
  {"ts": 1778649310, "chain": "arbitrum", "rtt_ms": 2717, "risk": 0.95, "stall_flag": true},
  {"ts": 1778649318, "chain": "arbitrum", "rtt_ms": 1843, "risk": 0.90, "stall_flag": true},
  {"ts": 1778649330, "chain": "arbitrum", "rtt_ms": 921,  "risk": 0.78, "stall_flag": true},
  {"ts": 1778649348, "chain": "arbitrum", "rtt_ms": 287,  "risk": 0.55, "stall_flag": true},
  {"ts": 1778649352, "chain": "arbitrum", "rtt_ms": 18,   "risk": 0.06, "stall_flag": false, "note": "RECOVERY"}
]

Event summary:

• Normal baseline: ~14ms (P99 ≈ 85ms)
• Peak during stall: 3081ms — 36x above P99 baseline
• Duration: 47 seconds (05:15:05 → 05:15:52 UTC)
• First kernel-visible: 05:15:06 UTC
• First visible to public RPC layer: 05:15:53 UTC
• Gap: 47 seconds of stall that application-layer health checks did not see

---

On-Chain Evidence

Keeper/solver transaction failed at block 462289970 (05:15:11 UTC — 6 seconds into the stall):

TX:     0x93fc225dc2aded30eb0c51185467b547ccb9d989bb83db3feb80538a2f9b924c
Block:  462289970
Status: FAILED (0x0)
From:   0x0587a02effe763c8b8e686cac7ac0dab20a36d35
To:     0x93c30e6e09f698cfea99b09b5166ed90cfaedd17
Gas:    0.03 Gwei | Logs: 0 (reverted before any state change)

Verify on Arbiscan

0.03 Gwei + zero logs = automated keeper that submitted into the stall and reverted on stale state. Public RPCs showed the sequencer as healthy at the time of submission.

---

Why This Matters for SUAVE / Relay Routing

If a Kettle or relay is routing based on public RPC health signals, it remains blind to this 47-second window. During that window:

1. Bundles are submitted against a sequencer that is TCP-reachable but not processing
2. Inclusions fail (TX above is one example — there were others)
3. Solver reputation degrades on failures that were not the solver's fault

A kernel-level TCP probe closes this gap. The kprobe fires at the network stack — it cannot be fooled by a sequencer that accepts connections but queues transactions internally.

---

What I Captured (Not All Shared Here)

The probe ran 5 hours 32 minutes continuously across 4 chains (03:57–09:29 UTC), capturing ~132,000 probe records. This issue shows a small excerpt.

The same session captured:

• ZKSync: 20+ stall events, peak 2928ms at 08:26 UTC
• Optimism: 5+ events, peak 1210ms
• Base: 0 stall events — fully stable during Azul hardfork activation

The full .jsonl session dump, the probe architecture, and the correlation methodology are not public. If this data is useful for SUAVE routing research or relay health oracle design, I'm happy to discuss directly.

---

— Kent
aleksandrkent64@gmail.com

[kant19801201behax5]

Update (May 14, 2026): Real-time Telegram alert bot is now live

Phoenix Zero RTT Oracle is now broadcasting live sequencer latency alerts via Telegram:

Bot: @Phoenix_Node_Alex_Bot
WebSocket stream: wss://rtt.phoenix-ai.work/ws

Alert format (firing in real time on any stall event):
`
🔵 SLOW CREEP — ARBITRUM
P99: 433ms | P95: 310ms (degrading)
📡 rtt.phoenix-ai.work

🟣 SEQUENCER STALL — ZKSYNC
RTT: 929ms | P99: 929ms | P95: 929ms
⚠️ Sequence-Aware Routing: pause strategies on Zksync
📡 rtt.phoenix-ai.work
`

Two alert types:

• 🔵 SLOW CREEP — P99 across 20-sample window exceeds 400ms (sustained degradation, like the Azul hardfork pattern on May 13)
• 🟣 SEQUENCER STALL — instant RTT spike >400ms (sudden event, catchable within one probe cycle = 2s)

All four chains monitored continuously (Base, Arbitrum, Optimism, ZKSync). The RTT spike data from May 13 in the original post was captured by this same infrastructure — the alert bot is the real-time consumer of that data stream.

Contact: aleksandrkent64@gmail.com

[kant19801201behax5]

Phoenix Zero RTT Oracle — 27h incident data from May 17-18

Sharing live measurement data from our sequencer RTT monitoring system covering the May 17 MEV event.

Setup: 2-second HTTP eth_blockNumber probes across Base, Arbitrum, Optimism, ZKSync. Revert ratio via eth_getBlockReceipts on each block.

May 17-18 stats (195,876 measurements, 27h window)

Chain	Avg P99	Max P99	Stall events (≥5s)
Arbitrum	76ms	5000ms	20
Optimism	232ms	1629ms	0
Base	317ms	5064ms	62
ZKSync	474ms	5099ms	2,500

Base revert ratio: avg 6.84%, peak 61.36%, 58 events above 30% threshold.

May 17 23:29–23:42 UTC event (key finding)

P99 RTT spiked from 52ms → 739ms at 23:29:43 UTC.
Base revert ratio crossed 0.50 at 23:30:10 UTC — 27 seconds later.

Full sequencer stall at 23:41:52 UTC (P99 = 5000ms timeout).

This 27-second leading indicator is consistent across multiple stress events in the dataset.

ZKSync anomaly

2,500 stall events in 27 hours = 5.1% of measurement windows. Sustained degradation pattern, not a single event spike. Worth cross-referencing with ZKSync sequencer logs from the same window.

---

Not a sales pitch — sharing the data in case it's useful for MEV research. Happy to export the raw JSONL for specific time windows.

Contact: aleksandrkent64@gmail.com