Skip to content

feat: make gateway rate limiter store pluggable#357

Merged
greatest0fallt1me merged 4 commits into
CalloraOrg:mainfrom
Agbasimere:feature/shared-rate-limiter-store-v3
May 28, 2026
Merged

feat: make gateway rate limiter store pluggable#357
greatest0fallt1me merged 4 commits into
CalloraOrg:mainfrom
Agbasimere:feature/shared-rate-limiter-store-v3

Conversation

@Agbasimere
Copy link
Copy Markdown
Contributor

Closes: #305

Summary

  • replace the gateway rate limiter with a pluggable store-backed implementation
  • keep InMemoryRateLimiter as the default fallback for local and single-instance use
  • add a PostgreSQL-backed shared token bucket store for multi-instance deployments behind a load balancer
  • wire limiter selection through config and startup
  • document the new env vars and shared-store behavior
  • add targeted tests for refill, exhaustion, malformed persisted data, and concurrency behavior

What Changed

Rate limiter

  • extracted a RateLimiterStore abstraction in src/services/rateLimiter.ts
  • preserved the existing token-bucket semantics and RateLimitResult shape
  • kept InMemoryRateLimiter as the default implementation
  • added PostgresRateLimiterStore with:
    • lazy table creation
    • transactional bucket updates
    • safe table-name validation
    • per-key row locking semantics for shared-instance correctness

Wiring

  • added config and env support for:
    • RATE_LIMIT_MAX_REQUESTS
    • RATE_LIMIT_WINDOW_MS
    • RATE_LIMIT_STORE
    • RATE_LIMIT_PG_TABLE
  • wired the configured limiter in src/index.ts
  • updated gateway and proxy routes to await async limiter checks:
    • src/routes/gatewayRoutes.ts
    • src/routes/proxyRoutes.ts

Docs

  • updated .env.example
  • documented shared gateway rate limiting and the new env vars in README.md

Behavior

  • token bucket refill behavior is preserved
  • retryAfterMs behavior is preserved
  • when RATE_LIMIT_STORE=postgres, limits are shared across instances
  • when persistence is not configured, the app falls back to in-memory buckets

Testing

  • npm run typecheck
  • npm run lint
  • npx jest --runInBand --forceExit src/services/rateLimiter.test.ts
  • npx jest --runInBand --forceExit src/config/index.test.ts src/config/env.test.ts src/routes/gatewayRoutes.test.ts
  • npx jest --runInBand --forceExit --coverage --collectCoverageFrom=src/services/rateLimiter.ts src/services/rateLimiter.test.ts

Coverage Notes

src/services/rateLimiter.ts

  • Statements: 100%
  • Branches: 95.23%
  • Functions: 100%
  • Lines: 100%

Notes

  • the limiter interface is now async, so route call sites now await check(...)
  • PostgreSQL-backed buckets use a shared backing table for cross-instance coordination
  • the shared bucket table is created lazily on first use
  • npm test -- src/services/rateLimiter.test.ts hit a local Jest worker spawn EPERM issue in this environment, so verification used the equivalent --runInBand form

@Agbasimere Agbasimere force-pushed the feature/shared-rate-limiter-store-v3 branch from 440923f to 1d79b7b Compare May 27, 2026 11:42
@Agbasimere Agbasimere force-pushed the feature/shared-rate-limiter-store-v3 branch 2 times, most recently from c8d313a to ac6a751 Compare May 27, 2026 13:45
@greatest0fallt1me
Copy link
Copy Markdown
Contributor

Merged via direct push to main (admin)

@greatest0fallt1me greatest0fallt1me merged commit d1685be into CalloraOrg:main May 28, 2026
1 check failed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Replace in-memory InMemoryRateLimiter token buckets with a shared persistent store for multi-instance deployments

2 participants