Skip to content

stacks/profile 429s: add Hiro client resilience (retry/backoff/Retry-After/cache) + verify prod key #124

@whoabuddy

Description

@whoabuddy

Summary

The /stacks/profile/:address endpoint 429s on roughly an hourly cadence in prod (x402-api-host). A single hourly request shouldn't be able to 429 a properly-keyed client — so this is a symptom of two real gaps: the Hiro client has no rate-limit resilience, and the prod Hiro key situation needs verification.

Evidence (worker-logs, last 72h, prod)

  • 50 errors in the window, all on one path: /stacks/profile/SP2PABAF9FTAJYNFZH93XENAJ8FVY99RRM50D2JG9, timestamps clustered at HH:00:0X (the hourly cron test).
  • Errors cascade from Hiro 429: Hiro API error (Failed to fetch block/balance/account info: 429) → Failed to fetch current block / Failed to fetch BNS names / account info / balance.
  • Payment settlement failed with errorReason: conflicting_nonce (separate concern, see Scheduled job sends concurrent settlements causing batch nonce conflicts #86).

Root cause

  1. No resilience in the Hiro clientsrc/services/hiro.ts uses raw fetch() with no retry, no backoff, no Retry-After handling, no rate-limit header reading, and no caching. Any 429 fails the whole request immediately.
  2. Burst fan-outsrc/endpoints/stacks/profile.ts fires 3–4 Hiro calls per request (BNS lookup + Promise.all of getAccountBalance / getAccountInfo / getCurrentBlock), hitting the per-second limiter harder than a serial call.
  3. Retry amplification — when a call 429s the endpoint returns 500, and the cron test harness retries the whole request up to 3×, turning one profile test into ~16 Hiro calls and self-inflicting more 429s.
  4. Key/budget question — the client supports x-api-key from c.env.HIRO_API_KEY, but .env carries a shared dev key. Whether the prod secret is set, and whether it shares a budget with landing-page (whose competition sweep is draining the same Hiro account — see landing-page companion issue), is unverified.

Proposal

  • Verify prod HIRO_API_KEY secret is set (wrangler secret list) and decide shared-vs-separate key/budget vs landing-page. (fast, high-leverage — may resolve most of this on its own)
  • Adopt the resilient Hiro-fetch pattern already standardized in landing-page (lib/stacks-api-fetch.ts): 429-specific exponential backoff, honor Retry-After, read x-ratelimit-*-stacks-month headers, emit stacksApi.* telemetry. Ideal long-term: extract a shared package.
  • Add a short cache for profile sub-calls — block height ~60s, balance/account ~10–30s, BNS ~1h.
  • Stop the test harness from retrying 429-origin 500s (avoid the amplification loop).

Reference

  • landing-page/lib/stacks-api-fetch.ts — the standard to mirror (Retry-After, monthly-quota header parsing, stacksApi.* events)

Priority: Medium (low volume, single path, not user-facing — but it's masking a missing-resilience defect)


🤖 Generated with Claude Code

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions