feat: add benchmarks api for remote management of runs#8
Open
cbermudez97 wants to merge 10 commits intomainfrom
Open
feat: add benchmarks api for remote management of runs#8cbermudez97 wants to merge 10 commits intomainfrom
cbermudez97 wants to merge 10 commits intomainfrom
Conversation
…e update paths, and network options
…egration - Added FastAPI application structure with routes for managing benchmark runs and scenarios. - Implemented database models for run tracking and API token management. - Introduced authentication via Bearer tokens for API access. - Created endpoints for submitting runs, listing runs, and retrieving run details. - Added functionality for downloading run outputs as ZIP archives. - Integrated metrics parsing from K6 summary files for detailed performance insights. - Established a background worker for processing benchmark runs asynchronously. - Updated CLI commands to support new API features and improved logging capabilities.
- Added a health check endpoint to monitor API status and database connectivity. - Implemented functionality to cancel queued benchmark runs, updating their status accordingly. - Introduced a new response model for run status retrieval. - Updated token management to track the last used timestamp upon successful authentication. - Added tests for new endpoints and functionalities to ensure reliability.
There was a problem hiding this comment.
Pull request overview
This PR introduces a FastAPI-based “benchmark queue” API that allows remote clients to list scenarios, submit benchmark runs with overrides, poll run status, and download run outputs. It also adds supporting CLI commands (serve API, manage API tokens) and a new pytest suite covering the API surface.
Changes:
- Add FastAPI app factory, auth, DB layer (SQLite/SQLAlchemy), background worker, and API routes for health/scenarios/runs.
- Add CLI subcommands for serving the API and managing tokens, and reorganize CLI entrypoints.
- Add API test suite (fixtures + endpoint tests) and adjust K6 summary-export thresholds to capture group metrics.
Reviewed changes
Copilot reviewed 27 out of 40 changed files in this pull request and generated 12 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/conftest.py | Shared pytest fixtures for DB/app/client, auth headers, and seeded runs |
| tests/api/test_scenarios.py | Tests for GET /scenarios auth + response shape |
| tests/api/test_runs.py | Tests for run submission, listing, status, cancel, and download |
| tests/api/test_health.py | Tests for unauthenticated health endpoint and queue counters |
| tests/api/test_auth.py | Tests for bearer token validation, last_used_at update, revocation |
| tests/api/init.py | Marks tests.api package |
| tests/init.py | Marks tests package |
| src/expb/payloads/executor/services/k6.py | Adjust K6 thresholds to export group-tagged duration sub-metrics |
| src/expb/cli/version/_version.py | New location for generated version/commit info |
| src/expb/cli/version/init.py | Update version command to import from new version module |
| src/expb/cli/send_payloads.py | New CLI command to send payloads/FCUs to an engine endpoint |
| src/expb/cli/generate_payloads.py | New CLI command to generate payload requests over a block range |
| src/expb/cli/execute_scenarios.py | New CLI command to execute multiple scenarios (optionally filtered) |
| src/expb/cli/execute_scenario.py | New CLI command to execute a single scenario |
| src/expb/cli/compress_payloads.py | New CLI command to compress payload transactions |
| src/expb/cli/api/tokens.py | New CLI subcommands to add/list/revoke API tokens |
| src/expb/cli/api/serve.py | New CLI command to run the FastAPI server via uvicorn |
| src/expb/cli/api/init.py | Registers api CLI subgroup and its subcommands |
| src/expb/cli/init.py | Central CLI registration for all subcommands |
| src/expb/api/worker.py | Background worker thread to execute queued runs sequentially |
| src/expb/api/schemas/runs.py | Pydantic request/response schemas for runs and K6 metrics |
| src/expb/api/schemas/init.py | Marks schemas package |
| src/expb/api/routes/scenarios.py | GET /scenarios route with overridable parameter metadata |
| src/expb/api/routes/runs.py | CRUD-style run routes: submit/list/get/status/cancel/download |
| src/expb/api/routes/health.py | Unauthenticated /health endpoint with DB + queue counters |
| src/expb/api/routes/init.py | Marks routes package |
| src/expb/api/metrics.py | Parser for K6 summary-export JSON into stored metric groups |
| src/expb/api/dependencies.py | FastAPI dependency providing per-request DB sessions |
| src/expb/api/db/models.py | SQLAlchemy models for runs and API tokens |
| src/expb/api/db/engine.py | SQLite engine/session initialization (WAL, multithread) |
| src/expb/api/db/init.py | Marks db package |
| src/expb/api/auth.py | Bearer token verification dependency with last_used_at stamping |
| src/expb/api/app.py | FastAPI app factory wiring DB, scenarios config, worker, and routers |
| src/expb/api/init.py | Marks api package |
| src/expb/_version.py | Remove old top-level version module |
| src/expb/init.py | Switch entrypoint to new expb.cli Typer app |
| pyproject.toml | Add FastAPI/SQLAlchemy/uvicorn deps and pytest dev deps/config |
| hatch_build.py | Update version-file write path to new CLI version module |
| .gitignore | Ignore .remote-test.env |
Comments suppressed due to low confidence (1)
pyproject.toml:45
filterwarningsentries must be valid pytest warning-filter strings (e.g.ignore::DeprecationWarning:module). The"hatchling>=1.29.0"entry is a dependency spec and will be ignored/raise parsing issues. Remove it or replace it with the intended warning filter.
[tool.pytest.ini_options]
filterwarnings = ["ignore::DeprecationWarning:websockets", "hatchling>=1.29.0"]
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+147
to
+149
| def queued_run(db_path: Path) -> Run: | ||
| init_db(db_path) | ||
| db = get_session() |
There was a problem hiding this comment.
This fixture also calls init_db(db_path), contributing to repeated global DB reinitialization across fixtures. Consolidating DB init into one fixture will make the test suite more deterministic.
Comment on lines
+174
to
+176
|
|
||
| init_db(db_path) | ||
| db = get_session() |
There was a problem hiding this comment.
This is another repeated init_db(db_path) call. Since init_db replaces the global engine/sessionmaker, repeated calls across fixtures can lead to hard-to-debug cross-test interactions.
Comment on lines
+121
to
+123
| """Insert an active API token into the DB and return the raw value.""" | ||
| init_db(db_path) | ||
| db = get_session() |
There was a problem hiding this comment.
Calling init_db(db_path) here can reinitialize the global engine/sessionmaker while a TestClient is active (depending on fixture ordering). Prefer a single shared fixture that initializes the DB once per test (or make init_db idempotent).
…agement improvements - Introduced ScenarioOverrides model for optional per-run configuration adjustments. - Updated SubmitRunRequest to include scenario overrides. - Improved token verification logic to raise appropriate HTTP status codes. - Refined the BenchmarkWorker to handle scenario execution with applied overrides. - Adjusted database models to ensure accurate timestamp handling. - Cleaned up deprecated filter warnings in configuration files.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 27 out of 40 changed files in this pull request and generated 5 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- Changed the type of the status field in the Run model to RunStatus for better type safety. - Modified the list_runs function to accept RunStatus as a filter for run status. - Enhanced the download_run_output function to use a temporary file for ZIP creation, reducing memory usage. - Updated tests to reflect changes in the structure of overrides and scenario parameters.
- Updated the restart policy for the K6 container from "unless-stopped" to "no" to ensure that the container does not restart after the benchmark finishes, preventing the overwriting of metrics in k6-summary.json.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.