Prometheus: Add README for prometheus-test #8
Conversation
WalkthroughA comprehensive Prometheus Test Framework has been introduced, including installation instructions, configuration files, test step definitions, and a test runner with environment and cryptographic utilities. The framework supports test execution, state management, and data handling. Additionally, a detailed security and quality audit report has been added, identifying vulnerabilities and suggesting improvements. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant TestRunner
participant TestEnvironment
participant Worker
participant MongoDB
participant DataManager
User->>TestRunner: Run test(s)
TestRunner->>MongoDB: Check/reset state
TestRunner->>TestEnvironment: Initialize environment
TestEnvironment->>Worker: Start workers
loop For each test step
TestRunner->>DataManager: Prepare test data
TestRunner->>Worker: Execute test step
Worker-->>TestRunner: Return result
TestRunner->>MongoDB: Save test state
end
TestRunner->>TestEnvironment: Cleanup environment
TestEnvironment->>Worker: Stop workers
TestRunner-->>User: Report results
Poem
Tip ⚡️ Faster reviews with caching
Enjoy the performance boost—your workflow just got faster. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (4)
SECURITY_AUDIT_Prometheus-beta.md (4)
37-61: Include missing imports and adjust file mode
The suggestedload_keypairsnippet is not self-contained—add the required imports (import json,import logging,import base58,from nacl.signing import SigningKey, andfrom typing import Tuple) at the top. Also, sincejson.loadexpects text, open the file in'r'mode rather than'rb'.
82-93: Clarify signature generation snippet and imports
Thecreate_signatureexample should includeimport jsonandimport base58alongsidefrom nacl.signing import SigningKeyandfrom nacl.encoding import Base64Encoder. Consider returning only the Base64‐encoded signature (without concatenating payload) or clearly document the combined format to avoid confusion.
1-20: Enhance Table of Contents linkability
Convert the bare TOC entries into GitHub‐style anchors, e.g.:
[Security Vulnerabilities](#security-vulnerabilities)[Code Quality Issues](#code-quality-issues)
This will improve navigation in long reports.
156-164: Clarify rating legend and next steps
The “🟨 Cautiously Acceptable” security rating could benefit from a brief legend or color key at the top, and you may want to outline a prioritized roadmap for addressing the top findings.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
⛔ Files ignored due to path filters (4)
.kno/embedding_SBERTEmbedding_1746699565308_b21fabe/4cf3883a-134b-40a5-a579-d09e34d56c29/data_level0.binis excluded by!**/*.bin.kno/embedding_SBERTEmbedding_1746699565308_b21fabe/4cf3883a-134b-40a5-a579-d09e34d56c29/header.binis excluded by!**/*.bin.kno/embedding_SBERTEmbedding_1746699565308_b21fabe/4cf3883a-134b-40a5-a579-d09e34d56c29/length.binis excluded by!**/*.bin.kno/embedding_SBERTEmbedding_1746699565308_b21fabe/4cf3883a-134b-40a5-a579-d09e34d56c29/link_lists.binis excluded by!**/*.bin
📒 Files selected for processing (2)
.kno/chunk_review.txt(1 hunks)SECURITY_AUDIT_Prometheus-beta.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
.kno/chunk_review.txt
[misspelling] ~140-~140: This word is normally spelled with a hyphen.
Context: ... MongoDB, you may need to do additional post processing (e.g. adding UUIDs). You can define a p...
(EN_COMPOUNDS_POST_PROCESSING)
[uncategorized] ~152-~152: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...f you have an .env file in your agent's top level folder (for API keys, etc), those envir...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
[style] ~152-~152: In American English, abbreviations like “etc.” require a period.
Context: ...agent's top level folder (for API keys, etc), those environment variables will be a...
(ETC_PERIOD)
[uncategorized] ~152-~152: Use a comma before ‘or’ if it connects two independent clauses (unless they are closely connected and short).
Context: ...nt to add testing specific ENV variables or you need to override any values from yo...
(COMMA_COMPOUND_SENTENCE)
[uncategorized] ~152-~152: “you” seems less likely than “your” (belonging to you).
Context: ...or you need to override any values from you main .env, you can add a second .env in...
(AI_HYDRA_LEO_CP_YOU_YOUR)
[uncategorized] ~232-~232: Loose punctuation mark.
Context: ...e2e [--reset] ``` Options: - --reset: Force reset of all databases before run...
(UNLIKELY_OPENING_PUNCTUATION)
[style] ~236-~236: Consider a more concise word here.
Context: ... the remote MongoDB instance (if using) in order to keep the state in sync. === File: .git...
(IN_ORDER_TO_PREMIUM)
[duplication] ~247-~247: Possible typo: you repeated a word.
Context: ... pycache .pytest_cache .pypirc *.db test test_state.json task_flow.egg-info example_r...
(ENGLISH_WORD_REPEAT_RULE)
[duplication] ~283-~283: Possible typo: you repeated a word.
Context: ...epository directories repos/ # Ignore Data data/* venv **/venv/ === File: .prettier...
(ENGLISH_WORD_REPEAT_RULE)
[uncategorized] ~338-~338: Did you just mean “,” or “)”?
Context: ...0.0", ], python_requires=">=3.8", ) === File: prometheus_test/init.py ...
(COMMA_PERIOD)
|
|
||
| === File: README.md === | ||
|
|
||
| -- Chunk 1 -- | ||
| // /app/repos/repo_3/repos/repo_0/repos/repo_0/repos/repo_0/repos/repo_0/repos/repo_0/repos/repo_0/repos/repo_0/repos/repo_0/repos/repo_0/README.md:1-150 | ||
| # Prometheus Test Framework Usage Guide | ||
|
|
||
| ## Getting Started | ||
|
|
||
| ### Installation | ||
|
|
||
| ```bash | ||
| pip install -e test-framework/ | ||
| ``` | ||
|
|
||
| ### Basic Structure | ||
|
|
||
| A test implementation consists of three main components: | ||
|
|
||
| 1. Configuration Files | ||
| 2. Test Steps Definition | ||
| 3. Test Runner Script | ||
|
|
||
| ## Creating a Test | ||
|
|
||
| ### 1. Configuration | ||
|
|
||
| #### Test Configuration (config.yaml) | ||
|
|
||
| ```yaml | ||
| # Test Configuration | ||
| task_id: "your_task_id" # Task identifier | ||
| base_port: 5000 # Base port for worker servers, optional | ||
| max_rounds: 3 # Maximum test rounds, optional | ||
|
|
||
| # Paths | ||
| data_dir: data # Test data directory, optional. defaults to the /data dir within your tests folder | ||
| workers_config: workers.json # Worker configuration, relative to tests directory | ||
|
|
||
| # MongoDB Configuration (if needed) | ||
| mongodb: | ||
| database: your_database_name | ||
| collections: | ||
| collection_name: | ||
| data_file: data.json # Relative to data_dir | ||
| required_count: 1 # Minimum required documents | ||
| ``` | ||
|
|
||
| #### Worker Configuration (workers.json) | ||
|
|
||
| ```json | ||
| { | ||
| "worker1": { | ||
| "port": 5001, | ||
| "env": { | ||
| "WORKER_ID": "worker1", | ||
| "OTHER_ENV": "value" | ||
| } | ||
| }, | ||
| "worker2": { | ||
| "port": 5002, | ||
| "env": { | ||
| "WORKER_ID": "worker2" | ||
| } | ||
| } | ||
| } | ||
| ``` | ||
|
|
||
| ### 2. Defining Test Steps | ||
|
|
||
| Create a `steps.py` file to define your test sequence: | ||
|
|
||
| ```python | ||
| from prometheus_test import TestStep | ||
|
|
||
| steps = [ | ||
| TestStep( | ||
| name="step_name", # Unique step identifier | ||
| description="Step description", # Human-readable description | ||
| prepare=your_prepare_function, # Setup function | ||
| execute=your_execute_function, # Main execution function | ||
| worker="worker_name", # Worker that executes this step | ||
| ), | ||
| # Add more steps... | ||
| ] | ||
| ``` | ||
|
|
||
| If you need to add extra parameters when calling prepare or execute functions you can `partial` from `functools` | ||
|
|
||
| ```py | ||
| from functools import partial | ||
|
|
||
| ... | ||
| TestStep( | ||
| name="step_name", | ||
| description="Step description", | ||
| prepare=your_prepare_function, | ||
| execute=partial(your_execute_function, extra_parameter=value), | ||
| worker="worker_name", | ||
| ), | ||
| ... | ||
|
|
||
| ``` | ||
|
|
||
| ### 3. Test Runner Script | ||
|
|
||
| Create a main test script (e.g., `e2e.py`) that sets up and runs your test sequence: | ||
|
|
||
| ```python | ||
| from pathlib import Path | ||
| from prometheus_test import TestRunner | ||
| import dotenv | ||
|
|
||
| # Load environment variables | ||
| dotenv.load_dotenv() | ||
|
|
||
| # Import your test steps | ||
| from .steps import steps | ||
|
|
||
| def main(): | ||
| # Create test runner with config from YAML | ||
| base_dir = Path(__file__).parent | ||
| runner = TestRunner( | ||
| steps=steps, | ||
| config_file=base_dir / "config.yaml", | ||
| config_overrides={ | ||
| "post_load_callback": your_callback_function # Optional | ||
| } | ||
| ) | ||
|
|
||
| # Run test sequence | ||
| runner.run(force_reset=False) | ||
|
|
||
| if __name__ == "__main__": | ||
| main() | ||
| ``` | ||
|
|
||
| ### 4. Post Load Callback | ||
|
|
||
| If you're loading data from JSON files into MongoDB, you may need to do additional post processing (e.g. adding UUIDs). You can define a post load callback in `e2e.py` which will be automatically executed after the MongoDB collections have been populated. | ||
|
|
||
| ```python | ||
| def post_load_callback(db): | ||
| """Modify database after initial load""" | ||
| for doc in db.collection.find(): | ||
| # Modify documents as needed | ||
| db.collection.update_one({"_id": doc["_id"]}, {"$set": {"field": "value"}}) | ||
| ``` | ||
|
|
||
| ### 5. ENV Variables | ||
|
|
||
| If you have an .env file in your agent's top level folder (for API keys, etc), those environment variables will be automatically loaded into your test script. If you want to add testing specific ENV variables or you need to override any values from you main .env, you can add a second .env in your tests/ directory, which will also be automatically loaded and overrides will be applied. | ||
|
|
There was a problem hiding this comment.
Remove internal AI artifact
.kno/chunk_review.txt contains AI chunking metadata and shouldn’t be versioned as part of the codebase. Remove this file (or add .kno/ to .gitignore) and replace it with a properly named README.md at the repository root.
🧰 Tools
🪛 LanguageTool
[misspelling] ~140-~140: This word is normally spelled with a hyphen.
Context: ... MongoDB, you may need to do additional post processing (e.g. adding UUIDs). You can define a p...
(EN_COMPOUNDS_POST_PROCESSING)
[uncategorized] ~152-~152: If this is a compound adjective that modifies the following noun, use a hyphen.
Context: ...f you have an .env file in your agent's top level folder (for API keys, etc), those envir...
(EN_COMPOUND_ADJECTIVE_INTERNAL)
[style] ~152-~152: In American English, abbreviations like “etc.” require a period.
Context: ...agent's top level folder (for API keys, etc), those environment variables will be a...
(ETC_PERIOD)
[uncategorized] ~152-~152: Use a comma before ‘or’ if it connects two independent clauses (unless they are closely connected and short).
Context: ...nt to add testing specific ENV variables or you need to override any values from yo...
(COMMA_COMPOUND_SENTENCE)
[uncategorized] ~152-~152: “you” seems less likely than “your” (belonging to you).
Context: ...or you need to override any values from you main .env, you can add a second .env in...
(AI_HYDRA_LEO_CP_YOU_YOUR)
1 participant
Prometheus: Add README for prometheus-test
Summary of Changes
This PR adds a README file for the prometheus-test repository.
Summary by CodeRabbit
New Features
Documentation