Skip to content

ci: concurrent sampling ci test#4209

Merged
kapishupadhyay22 merged 5 commits into
mainfrom
ci-sampling
May 21, 2026
Merged

ci: concurrent sampling ci test#4209
kapishupadhyay22 merged 5 commits into
mainfrom
ci-sampling

Conversation

@kapishupadhyay22
Copy link
Copy Markdown
Member

@kapishupadhyay22 kapishupadhyay22 commented May 21, 2026

Describe the changes that are made

Adds an end-to-end GitHub Actions pipeline that exercises the --enable-sampling flag of keploy record and asserts the documented bypass behaviour.

What runs on every PR and every push to main:

  1. Build keploy from the current branch.
  2. Check out the companion sampling-test sample app from keploy/samples-go.
  3. Start keploy record --enable-sampling=$K -c ./sampling-test.
  4. Wait for the app on :8080, then run curl.sh to fire $N concurrent GET /work requests, each on its own TCP connection. The handler sleeps HANDLER_DELAY_MS (default 500 ms) so the burst overlaps inside the proxy and the slot semaphore is genuinely contended.
  5. Drain captures, send SIGINT to keploy.
  6. Inspect ./keploy/test-set-0/tests/ and assert:
    • every curl returned 2xx (the bypass path must never break clients — feedback_no_client_errors), and
    • $K ≤ captured_test_cases < $N (sampling actually gated some traffic, but not all of it).

Defaults: K=5, N=20, handler delay 500 ms. All overridable via workflow_dispatch inputs (sampling_limit, total_requests, samples_go_ref).

Files added:

  • .github/workflows/sampling-test.yml — workflow definition; triggers on pull_request, push to main, and workflow_dispatch.
  • .github/workflows/test_workflow_scripts/sampling/run-sampling-test.sh — driver script (build sample, record, curl burst, SIGINT, assert).

The companion sample app lives in keploy/samples-go#232. The workflow currently pins samples_go_ref to go-sampling so the two PRs can be merged in either order; flip the default to main once the samples-go PR is in.

Links & References

Closes: #4211

🔗 Related PRs

🐞 Related Issues

📄 Related Documents

  • NA

What type of PR is this? (check all applicable)

  • 📦 Chore
  • 🍕 Feature
  • 🐞 Bug Fix
  • 📝 Documentation Update
  • 🎨 Style
  • 🧑‍💻 Code Refactor
  • 🔥 Performance Improvements
  • ✅ Test
  • 🔁 CI
  • ⏩ Revert

Added e2e test pipeline?

  • 👍 yes — this PR is the e2e pipeline

Added comments for hard-to-understand areas?

  • 👍 yes — the driver script's count-and-assert block explains why we filter to get-work-*.yaml only

Added to documentation?

  • 📜 README.md
  • 📓 Wiki
  • 🙅 no documentation needed — the sample's README.md (in samples-go) explains how to reproduce locally

Are there any sample code or steps to test the changes?

  • 👍 yes, mentioned below

Reproduce locally:

# In keploy/samples-go (on the go-sampling branch)
cd sampling-test
go build -o sampling-test .

# In another terminal
sudo keploy record --enable-sampling=5 -c ./sampling-test &
TOTAL_REQUESTS=20 bash curl.sh

# Then SIGINT keploy and inspect
ls ./keploy/test-set-0/tests/   # expect 5x get-work-*.yaml + 1x get-health-1.yaml

Self Review done?

  • ✅ yes

Any relevant screenshots, recordings or logs?

Latest CI run on this branch — successful assertion (post-fix in commit 843d753):

captured /work test cases: 5
total recorded test cases (incl. /health probe): 6
sampling limit (K): 5
total /work requests (N): 20
✅ sampling assertion passed: 5 <= 5 < 20

curl-side, every client got a 2xx response:

curl summary: total=20 ok=20 non2xx=0 wait_failures=0

🧠 Semantics for PR Title & Branch Name

  • PR Title: ci: concurrent sampling ci test
  • Branch Name: ci-sampling

Additional checklist:

Signed-off-by: kapish <upadhyaykapish@gmail.com>
Copilot AI review requested due to automatic review settings May 21, 2026 08:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a dedicated GitHub Actions workflow to run an end-to-end concurrency sampling check against the samples-go/sampling-test app, validating that --enable-sampling gates recordings under concurrent load without breaking client responses.

Changes:

  • Introduces a new CI workflow (Sampling Test) to build Keploy, check out keploy/samples-go, and execute the sampling E2E test.
  • Adds a new bash harness that records with --enable-sampling, generates concurrent traffic, stops the recorder, and asserts captured test counts fall within expected bounds.
  • Uploads relevant artifacts (record log, recordings, curl results) on failure for debugging.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
.github/workflows/test_workflow_scripts/sampling/run-sampling-test.sh Adds the end-to-end sampling test driver script (build app, run record with sampling, fire concurrent traffic, assert capture bounds).
.github/workflows/sampling-test.yml Adds a new workflow to build Keploy, check out samples-go, run the sampling test script, and upload artifacts on failures.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@github-actions
Copy link
Copy Markdown

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run P50 P90 P99 RPS Error Rate Status
1 2.87ms 3.74ms 5.23ms 100.00 0.00% ✅ PASS
2 2.75ms 3.52ms 5.01ms 100.02 0.00% ✅ PASS
3 2.79ms 3.78ms 5.15ms 100.02 0.00% ✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

The previous count used `find -name 'test-*.yaml'`, but keploy names test
cases by HTTP method + path (get-work-N.yaml), so the find matched zero
files and the assertion failed even though sampling was working correctly
(K=5 /work captures out of N=20 concurrent requests).

Switch the find pattern to `get-work-*.yaml` so the readiness probe on
/health that lands in the same test-set doesn't inflate the count above
the sampling budget either. Also print the full file count separately for
debugging visibility.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: kapish <upadhyaykapish@gmail.com>
@github-actions
Copy link
Copy Markdown

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run P50 P90 P99 RPS Error Rate Status
1 2.6ms 3.24ms 4.64ms 100.02 0.00% ✅ PASS
2 2.38ms 3.1ms 4.39ms 100.02 0.00% ✅ PASS
3 2.68ms 3.46ms 4.72ms 100.02 0.00% ✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

@github-actions
Copy link
Copy Markdown

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run P50 P90 P99 RPS Error Rate Status
1 2.64ms 3.39ms 4.95ms 100.02 0.00% ✅ PASS
2 2.55ms 3.24ms 4.59ms 100.02 0.00% ✅ PASS
3 2.64ms 3.62ms 4.95ms 100.02 0.00% ✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

@github-actions
Copy link
Copy Markdown

🚀 Keploy Performance Test Results

Multi-Run Validation: Tests run 3 times, pipeline fails only if 2+ runs show regression.

Run P50 P90 P99 RPS Error Rate Status
1 2.71ms 3.33ms 4.77ms 100.02 0.00% ✅ PASS
2 2.64ms 3.28ms 4.52ms 100.02 0.00% ✅ PASS
3 2.64ms 3.41ms 4.73ms 100.02 0.00% ✅ PASS

Thresholds: P50 < 5ms, P90 < 15ms, P99 < 70ms, RPS >= 100 (±1% tolerance), Error Rate < 1%

Result: PASSED - Only 0 out of 3 runs failed (threshold: 2)

P50, P90, and P99 percentiles naturally filter out outliers

@kapishupadhyay22 kapishupadhyay22 merged commit 86311e3 into main May 21, 2026
258 of 268 checks passed
@kapishupadhyay22 kapishupadhyay22 deleted the ci-sampling branch May 21, 2026 14:07
@github-actions github-actions Bot locked and limited conversation to collaborators May 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants