Skip to content

feat: performance benchmarking module + baseline (KOJAK-68)#34

Merged
endrju19 merged 2 commits intomainfrom
feature/kojak-68-benchmarks
Apr 30, 2026
Merged

feat: performance benchmarking module + baseline (KOJAK-68)#34
endrju19 merged 2 commits intomainfrom
feature/kojak-68-benchmarks

Conversation

@endrju19
Copy link
Copy Markdown
Collaborator

Summary

Establishes a reproducible JMH-based performance benchmarking setup and captures the pre-optimization baseline for okapi. Provides a reference point against which all subsequent optimizations (KOJAK-70..79) will be measured and documented.

KOJAK-68 — Performance benchmarking module + baseline measurement

What's included

okapi-benchmarks module

  • JMH 1.37 via me.champeau.jmh Gradle plugin
  • Default config: fork=2, warmup=3, iter=5 for release-quality runs
  • Located in okapi-benchmarks/ (not published, like okapi-integration-tests)

Benchmarks

Class What it measures
KafkaThroughputBenchmark End-to-end fanout, real Postgres + real Kafka (Testcontainers), @Param batchSize ∈ {10, 50, 100}
HttpThroughputBenchmark End-to-end fanout, real Postgres + WireMock, @Param batchSize × httpLatencyMs ∈ {0, 20, 100}
DelivererMicroBenchmark Single-entry deliver() with MockProducer / WireMock — measures pure code overhead

Methodology touches

  • @OperationsPerInvocation(1000) so JMH ms/op directly reflects per-message cost
  • Scheduler is bypassed in benchmarks — we measure the processing pipeline's capacity, not polling cadence (which is a deployment-time knob)
  • Per-invocation @Setup truncates outbox and re-populates fresh PENDING entries

Documentation

  • benchmarks/README.md — methodology + how to run + caveats (WireMock in-JVM, localhost ≠ production)
  • benchmarks/results-baseline-2026-04.md — full baseline tables with interpretation

Baseline numbers (smoke run on M3 Max, JDK 25 LTS)

Kafka

batchSize msg/s
10 ~109
50 ~115
100 ~115

Flat across batchSize — confirms sync producer.send().get() blocking is the bottleneck.

HTTP

batchSize latencyMs msg/s
10 0 ~1,513
50 0 ~2,717
100 0 ~3,322
10 20 ~33
50 20 ~33
100 20 ~36
10 100 ~9
50 100 ~9
100 100 ~9

At non-zero latency, throughput is flat — proves sequential httpClient.send() blocking dominates. Library + DB ceiling visible only at latencyMs=0.

Microbenchmark

  • KafkaMessageDeliverer.deliver() with MockProducer: ~1.8M ops/s — pure code overhead is negligible

How to reproduce

# Quick smoke run (~10 min)
./gradlew :okapi-benchmarks:jmhJar
java -jar okapi-benchmarks/build/libs/okapi-benchmarks-jmh.jar -f 1 -wi 1 -i 2 -w 10s -r 15s

# Full release-quality run (~30 min)
./gradlew :okapi-benchmarks:jmh

Test plan

  • ./gradlew :okapi-benchmarks:jmhClasses compiles cleanly
  • ./gradlew :okapi-benchmarks:ktlintCheck passes
  • JMH discovers all 4 benchmarks (-l flag)
  • Smoke microbenchmark run completes successfully (DelivererMicroBenchmark.kafkaDeliver)
  • Smoke throughput run completes for both Kafka and HTTP transports across full @Param matrix
  • Results captured in benchmarks/results-baseline-2026-04.md

Notes for reviewers

  • WireMock caveat: in-JVM Jetty servlet adds ~0.3 ms overhead per request. Documented in README. Future optimization (parked in KOJAK-69 ideas backlog): swap to Square's MockWebServer for tighter measurements.
  • No CI integration yet: full benchmark run is ~30 min, not suitable for PR gate. Future task: regression detection via github-action-benchmark consuming the JSON output.
  • .log files are gitignored — verbose console output is not a useful artifact; structured JSON is sufficient for archival.

Add `okapi-benchmarks` module using JMH (1.37) via the me.champeau.jmh
Gradle plugin. Provides reproducible measurements of fanout throughput
across transports and configurations.

Benchmarks:
- KafkaThroughputBenchmark — real Postgres + real Kafka via Testcontainers
- HttpThroughputBenchmark — real Postgres + WireMock with @param httpLatencyMs
  injecting 0/20/100ms server-side delay (covers library-only ceiling vs
  realistic webhook scenarios)
- DelivererMicroBenchmark — single-entry deliver() with mocked I/O,
  measures pure code overhead (~1.8M ops/s for Kafka with MockProducer)

Methodology:
- @OperationsPerInvocation(1000) on throughput benchmarks so JMH ms/op
  directly reflects per-message cost (reciprocal = msg/s)
- @setup(Level.Trial) starts containers; @setup(Level.Invocation) populates
  fresh PENDING entries before each timed run
- Scheduler bypassed: `processor.processNext` called in a loop until drained
  to measure processing capacity, not polling cadence

Baseline results (smoke run, MacBook M3 Max, JDK 25, fork=1, warmup=1, iter=2):
- Kafka:      ~109-115 msg/s (flat across batchSize — confirms sync .get() bottleneck)
- HTTP @0ms:  ~1,500-3,300 msg/s (library + DB ceiling; WireMock has no real I/O)
- HTTP @20ms: ~33-36 msg/s (flat — sequential blocking dominates)
- HTTP @100Ms: ~9 msg/s (flat — close to theoretical 1000/100 = 10 msg/s)

These numbers establish a reference point for upcoming optimizations
(KOJAK-70..79) — every subsequent change re-runs the suite and documents
before/after in `benchmarks/results-postopt-*.md`.

For release-quality numbers with confidence intervals, run
`./gradlew :okapi-benchmarks:jmh` (default fork=2, warmup=3, iterations=5).
Adds a Performance section between Compatibility and Build summarizing
the KOJAK-68 baseline throughput across transports and batch sizes,
plus a link to the full benchmarks/ directory with methodology and
reproduction instructions.

Also extends the Build section with the JMH command for completeness.
@endrju19 endrju19 merged commit 1eb3a2e into main Apr 30, 2026
8 checks passed
@endrju19 endrju19 deleted the feature/kojak-68-benchmarks branch April 30, 2026 21:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant