Skip to content

Keynote 2 benchmark updates & refinements#4997

Open
bradleyshep wants to merge 31 commits into
masterfrom
bradley/keynote-benchmark-updates
Open

Keynote 2 benchmark updates & refinements#4997
bradleyshep wants to merge 31 commits into
masterfrom
bradley/keynote-benchmark-updates

Conversation

@bradleyshep
Copy link
Copy Markdown
Contributor

This PR builds on #4978

Description of Changes

Methodology refresh of templates/keynote-2/.

  • Pool size fix. Removed the poolMax = 1000 override on the pg-flavored RPC servers (postgres, cockroach, supabase). All now use node-pg's default of 10, matching bun-server.ts.
  • Convex retry loop removed. The 32-retry client-side loop in connectors/convex.ts was masking conflict-storms with a death spiral.
  • Convex package bumped from ^1.29 to ^1.37.
  • Hidden Postgres tuning removed from docker-compose.yml. Local pg keeps only the explicit default_transaction_isolation = 'serializable' override.
  • Per-second time-series sampling added to core/runner.ts. Each run emits a timeSeries array for warmup/decay/collapse charts.
  • Bench CLI accepts --alpha 0,1.5 (CSV), --runs N, and --prep-between-alphas, writing one JSON per (connector, alpha, run) tuple.
  • Helper scripts added: start-bench.sh, stop-bench.sh, check-bench.sh, bench-stats.py, plot-bench.py. Paths resolve relative to the script file.
  • README updated with current measurements, two PlanetScale rows (PS-2560 single-node EBS and M-15360 Metal HA), and isolation-level methodology notes.
  • Removed unused Docker files

API and ABI breaking changes

No engine API/ABI changes. Internal shape changes for the keynote template:

  • BenchOptions.alpha: numberalphas: number[]. New fields: runs, prepBetweenAlphas.
  • RunResult gained a required timeSeries field.
  • Output JSON filename is now test-1-<connector>-a<alpha>-<timestamp>.json (glob test-1-*.json still matches).

Expected complexity level and risk

3. Contained to templates/keynote-2/ but cross-cutting within it (runner, CLI, RPC servers, docker-compose, scripts, README).

Testing

Testing

Reviewer smoke test:

pnpm run prep
pnpm run bench --alpha 0,1.5 --connectors postgres_rpc --seconds 30 --runs 1
ls runs/test-1-postgres_rpc-a*.json   # expect 2 files
  • pnpm build passes
  • pnpm run bench --help lists the new flags
  • Reviewer: smoke sweep above produces 2 JSONs (one per alpha)
  • Reviewer: confirm --prep-between-alphas actually re-preps between alphas
  • Reviewer: confirm pnpm run demo still works

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants