feat: add agents configuration layer with backward-compatible api: migration#228
Merged
Merged
Conversation
Contributor
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds agent configuration models and resolution; integrates optional SystemConfig.agents with legacy api→agents migration; exposes per-conversation ChangesAgent Configuration and Selection System
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Tip 💬 Introducing Slack Agent: The best way for teams to turn conversations into code.Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.
Built for teams:
One agent for your entire SDLC. Right inside Slack. Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Actions performedReview triggered.
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/lightspeed_evaluation/core/models/agents.py`:
- Around line 116-123: The code mutates the input with data.pop("default") and
can overwrite previously collected agent entries by assigning agents = value
when encountering the "agents" key; to fix, avoid in-place mutation by reading
default_data = data.get("default", {}) or operate on a shallow copy (e.g., copy
= dict(data)), initialize agents as an empty dict before the loop, and when you
see the "agents" key merge its mapping into the agents dict instead of assigning
(and for flat entries where isinstance(value, dict) and "type" in value, insert
them into agents using a unique key), ensuring you use the existing agents dict
rather than replacing it (referencing variables default_data, agents, and the
loop that iterates over data.items()).
In `@src/lightspeed_evaluation/core/models/data.py`:
- Around line 405-409: The agent Field currently allows whitespace-only strings;
add validation to reject strings that are blank after trimming by adding a
Pydantic validator for the agent field (e.g., `@validator`("agent") or
`@field_validator`) that returns None if input is None but raises a ValueError
when str(value).strip() == "" (with a clear message like "agent must not be
empty or whitespace-only"); update any relevant error text and unit tests that
exercise the Agent resolution flow. Ensure the validator is implemented in the
same model class in data.py where the agent: Optional[str] Field is declared so
whitespace-only values are caught at model validation time.
In `@src/lightspeed_evaluation/core/models/system.py`:
- Around line 877-888: The current migration silently drops legacy API fields
because agent_fields only keeps keys in HttpApiAgentConfig.model_fields; detect
unexpected keys in api_data (for dicts and BaseModel dumps) by computing
unexpected = set(all_keys) - valid_fields - {"enabled"} and, if non-empty, raise
a clear error (e.g., ValueError) or otherwise fail fast with a message listing
the unexpected keys so callers know to update HttpApiAgentConfig or migrate
legacy fields; apply this check in the branches that use api_data, referencing
the variables api_data, valid_fields, agent_fields and
HttpApiAgentConfig.model_fields.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: f77ef196-b137-474d-b568-5fbac126e29a
📒 Files selected for processing (10)
src/lightspeed_evaluation/core/constants.pysrc/lightspeed_evaluation/core/models/__init__.pysrc/lightspeed_evaluation/core/models/agents.pysrc/lightspeed_evaluation/core/models/data.pysrc/lightspeed_evaluation/core/models/system.pysrc/lightspeed_evaluation/core/system/loader.pytests/unit/core/models/test_agents.pytests/unit/core/models/test_data.pytests/unit/core/models/test_system.pytests/unit/core/system/test_loader.py
rioloc
force-pushed
the
feat/agents-config-layer
branch
from
3f372ff to
c0b0678
Compare
Collaborator
Author
|
@coderabbitai review |
Contributor
✅ Actions performedReview triggered.
|
rioloc
force-pushed
the
feat/agents-config-layer
branch
6 times, most recently
from
cfcd43f to
5ec6b29
Compare
Contributor
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
src/lightspeed_evaluation/core/models/agents.py (1)
191-193:⚠️ Potential issue | 🟠 Major | ⚡ Quick winGuard nested
agentsmerge against non-mapping input.
agents.update(value)assumesvalueis a mapping. Malformed config (for example,agents: []) will throw a low-level exception during pre-validation instead of a clear config error.Suggested fix
for key, value in data.items(): if key == "agents": - agents.update(value) + if not isinstance(value, dict): + raise ConfigurationError( + "agents must be a mapping of agent names to definitions" + ) + agents.update(value) elif isinstance(value, dict) and "type" in value: agents[key] = value else: remaining[key] = value🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@src/lightspeed_evaluation/core/models/agents.py` around lines 191 - 193, The merge for the "agents" key currently calls agents.update(value) without ensuring value is a mapping, which will raise a low-level exception for non-mapping inputs (e.g., list); update the check in the same block that handles key == "agents" to first verify isinstance(value, dict) (or collections.abc.Mapping) and only call agents.update(value) when true, otherwise raise a clear config error (e.g., ValueError or ConfigError) indicating that "agents" must be a mapping; reference the existing "agents" variable and the key == "agents" branch to locate where to add the guard and the explicit error.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/integration/test_agents_config_compat.py`:
- Around line 48-53: The helper _find_csv currently allows multiple
"*_detailed.csv" files and returns the first, which can hide duplicate-output
regressions; update _find_csv to assert there is exactly one match (e.g. replace
the current truthy assert with a check like assert len(csvs) == 1 and include
the list of found files in the error message) and still return csvs[0];
reference the function name _find_csv and the local variable csvs to locate the
change.
- Around line 55-59: _sorting in _read_scores is currently only by
(conversation_group_id, turn_id) which is unstable when multiple metric rows
exist per turn; update the sort key in _read_scores to include additional
deterministic fields (e.g., metric name/metric_name and any scorer/score_id
fields) or fall back to a deterministic tuple of the remaining row values (e.g.,
tuple(r[k] for k in sorted(r.keys()) if k not in
("conversation_group_id","turn_id"))) so rows with the same group/turn are
ordered reproducibly; change the key lambda in _read_scores accordingly to use
those extra fields for stable ordering.
---
Duplicate comments:
In `@src/lightspeed_evaluation/core/models/agents.py`:
- Around line 191-193: The merge for the "agents" key currently calls
agents.update(value) without ensuring value is a mapping, which will raise a
low-level exception for non-mapping inputs (e.g., list); update the check in the
same block that handles key == "agents" to first verify isinstance(value, dict)
(or collections.abc.Mapping) and only call agents.update(value) when true,
otherwise raise a clear config error (e.g., ValueError or ConfigError)
indicating that "agents" must be a mapping; reference the existing "agents"
variable and the key == "agents" branch to locate where to add the guard and the
explicit error.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: f462bfd8-0aed-4f72-9fca-d34395fdf749
📒 Files selected for processing (15)
src/lightspeed_evaluation/core/constants.pysrc/lightspeed_evaluation/core/models/.agents.py.swpsrc/lightspeed_evaluation/core/models/__init__.pysrc/lightspeed_evaluation/core/models/agents.pysrc/lightspeed_evaluation/core/models/data.pysrc/lightspeed_evaluation/core/models/system.pysrc/lightspeed_evaluation/core/system/loader.pytests/integration/agents_config/eval_data.yamltests/integration/agents_config/system_agents.yamltests/integration/agents_config/system_legacy_api.yamltests/integration/test_agents_config_compat.pytests/unit/core/models/test_agents.pytests/unit/core/models/test_data.pytests/unit/core/models/test_system.pytests/unit/core/system/test_loader.py
✅ Files skipped from review due to trivial changes (2)
- src/lightspeed_evaluation/core/constants.py
- src/lightspeed_evaluation/core/system/loader.py
🚧 Files skipped from review as they are similar to previous changes (6)
- src/lightspeed_evaluation/core/models/init.py
- tests/unit/core/models/test_agents.py
- tests/unit/core/models/test_system.py
- tests/unit/core/system/test_loader.py
- src/lightspeed_evaluation/core/models/data.py
- tests/unit/core/models/test_data.py
rioloc
force-pushed
the
feat/agents-config-layer
branch
2 times, most recently
from
974ba94 to
a53c4a4
Compare
Contributor
There was a problem hiding this comment.
🧹 Nitpick comments (2)
tests/unit/core/models/test_data.py (2)
691-698: 💤 Low value
match="String should match pattern"is a brittle pydantic-internal message.The
match=string on line 693 couples the test to pydantic's internal error-message wording, which can change across pydantic minor/patch releases. The siblingtest_empty_agent_name_rejected(line 684) already omitsmatch=; consider doing the same here, or match on something stable like"agent"(the field name) or the regex pattern itself.♻️ Suggested fix
- with pytest.raises(ValidationError, match="String should match pattern"): + with pytest.raises(ValidationError):🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/core/models/test_data.py` around lines 691 - 698, The test_whitespace_agent_name_rejected currently asserts ValidationError with a brittle pydantic-internal message via match="String should match pattern"; update the pytest.raises call for EvaluationData(agent=" ") to either remove the match argument (like test_empty_agent_name_rejected) or use a stable matcher such as match="agent" or the actual regex pattern, so the test asserts a ValidationError for EvaluationData/TurnData without depending on pydantic's exact wording.
672-680: 💤 Low valueMissing coverage:
agent_configwithoutagent.All five new tests either set both fields or neither. It's worth adding a test (or a
pytest.mark.parametrizecase) that providesagent_configwithoutagentto document — and pin — the model's behaviour for that combination (accepted, rejected, or silently ignored).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/unit/core/models/test_data.py` around lines 672 - 680, Add a test case to explicitly cover the scenario where EvaluationData is instantiated with agent_config but without agent so the model behavior is pinned; update or add a new test (e.g., alongside test_agent_config_accepted or as a pytest.mark.parametrize) that constructs EvaluationData(conversation_group_id="cg1", agent_config={"timeout":1200}, turns=[TurnData(...)] ) and asserts the expected outcome (accepted and stored in data.agent_config, or raises a validation error) matching the intended behavior of the EvaluationData model; reference the EvaluationData class and the agent_config and agent fields when implementing the assertion so reviewers can see the documented behaviour.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In `@tests/unit/core/models/test_data.py`:
- Around line 691-698: The test_whitespace_agent_name_rejected currently asserts
ValidationError with a brittle pydantic-internal message via match="String
should match pattern"; update the pytest.raises call for EvaluationData(agent="
") to either remove the match argument (like test_empty_agent_name_rejected) or
use a stable matcher such as match="agent" or the actual regex pattern, so the
test asserts a ValidationError for EvaluationData/TurnData without depending on
pydantic's exact wording.
- Around line 672-680: Add a test case to explicitly cover the scenario where
EvaluationData is instantiated with agent_config but without agent so the model
behavior is pinned; update or add a new test (e.g., alongside
test_agent_config_accepted or as a pytest.mark.parametrize) that constructs
EvaluationData(conversation_group_id="cg1", agent_config={"timeout":1200},
turns=[TurnData(...)] ) and asserts the expected outcome (accepted and stored in
data.agent_config, or raises a validation error) matching the intended behavior
of the EvaluationData model; reference the EvaluationData class and the
agent_config and agent fields when implementing the assertion so reviewers can
see the documented behaviour.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: cfd5ea93-0c7a-4047-ae85-9912dae29791
📒 Files selected for processing (11)
src/lightspeed_evaluation/core/constants.pysrc/lightspeed_evaluation/core/models/.agents.py.swpsrc/lightspeed_evaluation/core/models/__init__.pysrc/lightspeed_evaluation/core/models/agents.pysrc/lightspeed_evaluation/core/models/data.pysrc/lightspeed_evaluation/core/models/system.pysrc/lightspeed_evaluation/core/system/loader.pytests/unit/core/models/test_agents.pytests/unit/core/models/test_data.pytests/unit/core/models/test_system.pytests/unit/core/system/test_loader.py
✅ Files skipped from review due to trivial changes (3)
- src/lightspeed_evaluation/core/system/loader.py
- src/lightspeed_evaluation/core/constants.py
- tests/unit/core/models/test_system.py
🚧 Files skipped from review as they are similar to previous changes (6)
- src/lightspeed_evaluation/core/models/data.py
- src/lightspeed_evaluation/core/models/system.py
- src/lightspeed_evaluation/core/models/init.py
- tests/unit/core/system/test_loader.py
- tests/unit/core/models/test_agents.py
- src/lightspeed_evaluation/core/models/agents.py
rioloc
force-pushed
the
feat/agents-config-layer
branch
from
a53c4a4 to
6ed659b
Compare
5 tasks
…gration Introduce a generic `agents:` configuration block in system.yaml that makes the HTTP API one agent type among potentially many (e.g., Kubernetes CRD agents for Openshift Agentic Lightspeed). The key design ensures zero breakage of existing configs and code: - New Pydantic models: AgentsConfig, AgentDefaultConfig, HttpApiAgentConfig in a new `core/models/agents.py` module - SystemConfig gains an `agents: Optional[AgentsConfig]` field alongside the existing `api:` field - A model_validator auto-migrates `api:` to `agents:` when `agents:` is absent, so all existing system.yaml files continue working unchanged - `api.enabled=True` maps to `default.agent="http_api"`; `api.enabled=False` maps to `default.agent=None` - The `api` field is preserved — all downstream code reading `config.api` continues to work without modification - EvaluationData gains `agent` and `agent_config` optional fields for per-conversation-group agent selection and config overrides - Config resolution follows a 3-level priority chain: eval_data.agent_config > agents.<name> > agents.default - ConfigLoader passes `agents` YAML data through to SystemConfig Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rioloc
force-pushed
the
feat/agents-config-layer
branch
from
6ed659b to
9555413
Compare
7 tasks
asamal4
reviewed
May 10, 2026
Collaborator
asamal4
left a comment
asamal4
left a comment
There was a problem hiding this comment.
Thank you !! PR looks good; very much aligned with the design doc..
I have couple of suggestions on the yaml structure. WDYT ?
Below will be aligned as per the eval data.
# Current
agents:
default:
agent: http_api
timeout: 300
retry: 3
http_api:
type: http_api
api_base: http://prod
# Proposed
agents:
default:
agent: api
agent_config:
timeout: 300
retry: 3
api:
type: http_api
api_base: http://prodagent null seems to be not very intuitive for live data disabled mode. Should we have enabled as part of agents.
# Current
agents:
default:
agent: null
# Proposed
agents:
enabled: false
default:
agent: api
rioloc
added a commit
to rioloc/lightspeed-evaluation
that referenced
this pull request
May 11, 2026
…nfig layer - Replace timeout/retry fields with generic agent_config dict on AgentDefaultConfig and HttpApiAgentConfig, implementing 3-level per-key merge (default < agent < eval_data) in resolve_agent_config - Move enabled from per-agent (HttpApiAgentConfig) to AgentsConfig level, add api_enabled property on SystemConfig for backward-compatible access - Move whitespace pattern validator from EvaluationData.agent to AgentDefaultConfig.agent where agent names are defined as identifiers - Update endpoint_type description to include infer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rioloc
added a commit
to rioloc/lightspeed-evaluation
that referenced
this pull request
May 11, 2026
…nfig layer - Replace timeout/retry fields with generic agent_config dict on AgentDefaultConfig and HttpApiAgentConfig, implementing 3-level per-key merge (default < agent < eval_data) in resolve_agent_config - Move enabled from per-agent (HttpApiAgentConfig) to AgentsConfig level, add api_enabled property on SystemConfig for backward-compatible access - Move whitespace pattern validator from EvaluationData.agent to AgentDefaultConfig.agent where agent names are defined as identifiers - Update endpoint_type description to include infer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rioloc
force-pushed
the
feat/agents-config-layer
branch
from
1a09355 to
c9e5428
Compare
Collaborator
Author
asamal4
reviewed
May 11, 2026
…nfig layer - Replace timeout/retry fields with generic agent_config dict on AgentDefaultConfig and HttpApiAgentConfig, implementing 3-level per-key merge (default < agent < eval_data) in resolve_agent_config - Move enabled from per-agent (HttpApiAgentConfig) to AgentsConfig level, add api_enabled property on SystemConfig for backward-compatible access - Move whitespace pattern validator from EvaluationData.agent to AgentDefaultConfig.agent where agent names are defined as identifiers - Update endpoint_type description to include infer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
rioloc
force-pushed
the
feat/agents-config-layer
branch
from
c9e5428 to
880f460
Compare
…i defaults _create_api_client now calls AgentsConfig.resolve_agent_config() to obtain the correct endpoint_type, api_base, and other settings from the default agent definition. Previously it used config.api directly, which fell back to legacy defaults (endpoint_type: streaming) even when agents were explicitly configured with endpoint_type: query. Widen APIClient config type to accept HttpApiAgentConfig alongside APIConfig since both share the same HttpApiBaseFields interface. Per-conversation agent overrides are not yet supported and will be addressed by the agent driver architecture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
asamal4
approved these changes
May 12, 2026
Collaborator
asamal4
left a comment
asamal4
left a comment
There was a problem hiding this comment.
Thanks !! LGTM..
Few action items for future:
- Let's add a warning when no api/agent is configured
- Bug fix for the override (not super critical, as this is the new capability. backward compatibility is to set right default agent)
- Documentation (Once we have the full integration)
Collaborator
@VladimirKadlec Agree. We will have to make sure that teams are using right config, before we remove current api: config completely. As of now no impact due to backward compatibility. @rioloc I edited the description to remove the doc ref.. May be I will add this later, after some clean-up. |
xmican10
pushed a commit
to xmican10/lightspeed-evaluation
that referenced
this pull request
May 13, 2026
…gration (lightspeed-core#228) * feat: add agents configuration layer with backward-compatible api: migration Introduce a generic `agents:` configuration block in system.yaml that makes the HTTP API one agent type among potentially many (e.g., Kubernetes CRD agents for Openshift Agentic Lightspeed). The key design ensures zero breakage of existing configs and code: - New Pydantic models: AgentsConfig, AgentDefaultConfig, HttpApiAgentConfig in a new `core/models/agents.py` module - SystemConfig gains an `agents: Optional[AgentsConfig]` field alongside the existing `api:` field - A model_validator auto-migrates `api:` to `agents:` when `agents:` is absent, so all existing system.yaml files continue working unchanged - `api.enabled=True` maps to `default.agent="http_api"`; `api.enabled=False` maps to `default.agent=None` - The `api` field is preserved — all downstream code reading `config.api` continues to work without modification - EvaluationData gains `agent` and `agent_config` optional fields for per-conversation-group agent selection and config overrides - Config resolution follows a 3-level priority chain: eval_data.agent_config > agents.<name> > agents.default - ConfigLoader passes `agents` YAML data through to SystemConfig Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * refactor: address PR lightspeed-core#228 review feedback on agents config layer - Replace timeout/retry fields with generic agent_config dict on AgentDefaultConfig and HttpApiAgentConfig, implementing 3-level per-key merge (default < agent < eval_data) in resolve_agent_config - Move enabled from per-agent (HttpApiAgentConfig) to AgentsConfig level, add api_enabled property on SystemConfig for backward-compatible access - Move whitespace pattern validator from EvaluationData.agent to AgentDefaultConfig.agent where agent names are defined as identifiers - Update endpoint_type description to include infer Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> * fix: resolve default agent config for API client instead of legacy api defaults _create_api_client now calls AgentsConfig.resolve_agent_config() to obtain the correct endpoint_type, api_base, and other settings from the default agent definition. Previously it used config.api directly, which fell back to legacy defaults (endpoint_type: streaming) even when agents were explicitly configured with endpoint_type: query. Widen APIClient config type to accept HttpApiAgentConfig alongside APIConfig since both share the same HttpApiBaseFields interface. Per-conversation agent overrides are not yet supported and will be addressed by the agent driver architecture. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Introduces a generic
agents:configuration block insystem.yamlthat makes the HTTP API one agent type among potentially many (e.g., Kubernetes CRD agents for Openshift Agentic Lightspeed). This is the foundational configuration layer — the driver interface and CRD implementation will follow in subsequent PRs.AgentsConfig,AgentDefaultConfig,HttpApiAgentConfigincore/models/agents.pySystemConfiggainsagents: Optional[AgentsConfig]with auto-migration fromapi:viamodel_validatorEvaluationDatagainsagentandagent_configoptional fields for per-conversation-group agent selectionagent_configmerge:default.agent_config<agents.<name>.agent_config<eval_data.agent_configenabledlives at theAgentsConfiglevel (not per-agent), withSystemConfig.api_enabledproperty for backward-compatible accessconfig.api.*access paths, YAML files, and tests continue working unchangedArchitecture
graph TD subgraph "system.yaml" API["api:<br/>(existing, preserved)"] AGENTS["agents:<br/>(new)"] end subgraph "SystemConfig" API_FIELD["api: APIConfig<br/>(unchanged)"] AGENTS_FIELD["agents: Optional[AgentsConfig]<br/>(new)"] MIGRATION["model_validator<br/>migrate_api_to_agents"] API_ENABLED["api_enabled property<br/>(agents.enabled → api.enabled fallback)"] end API -->|"loaded by ConfigLoader"| API_FIELD API -->|"auto-migrates when<br/>agents: absent"| MIGRATION MIGRATION --> AGENTS_FIELD AGENTS -->|"loaded by ConfigLoader"| AGENTS_FIELD subgraph "AgentsConfig" ENABLED["enabled: bool<br/>(agents-level toggle)"] DEFAULT["default:<br/>AgentDefaultConfig<br/>(agent selection + agent_config)"] HTTP["http_api:<br/>HttpApiAgentConfig"] FUTURE["kubernetes_crd:<br/>(future)"] end AGENTS_FIELD --> ENABLED AGENTS_FIELD --> DEFAULT AGENTS_FIELD --> HTTP AGENTS_FIELD -.-> FUTURE subgraph "eval_data.yaml" CG1["conversation_group<br/>(no agent = uses default)"] CG2["conversation_group<br/>agent: ols_api<br/>agent_config: {timeout: 1200}"] end subgraph "Config Resolution (per-key merge)" RESOLVE["resolve_agent_config()"] P1["1. default.agent_config<br/>(lowest priority)"] P2["2. agents.<name>.agent_config"] P3["3. eval_data.agent_config<br/>(highest priority)"] end CG2 --> RESOLVE P1 --> RESOLVE P2 --> RESOLVE P3 --> RESOLVE3-Level
agent_configMergeEach level provides an
agent_config: dictwith per-key merge in ascending priority order. Higher levels override matching keys; non-overlapping keys from lower levels survive:Result:
{timeout: 300, provider: azure, num_retries: 3}Backward Compatibility:
api:toagents:MigrationExisting
system.yamlfiles with only anapi:block require no changes. Amodel_validator(mode="before")onSystemConfigauto-migrates theapi:data into the newagents:structure at load time, while preserving the originalapifield for all downstream code.Existing config (unchanged, continues to work):
What the framework sees internally after auto-migration:
When
api.enabled: false:Key Design Decisions
enabledatAgentsConfiglevelapi_enabledproperty provides backward-compat accessagent_config: dictapifield on SystemConfigconfig.api.*— no breaking changesmodel_validator(mode="before")extra="forbid"on AgentsConfigmodel_validatoragentsdictFiles Changed
core/models/agents.pyAgentsConfig(withenabled),AgentDefaultConfig(withagent_config),HttpApiAgentConfig(withagent_config),resolve_agent_config()3-level mergecore/models/system.pyagentsfield, migration validators,api_enabledpropertycore/models/data.pyagent,agent_configfields toEvaluationDatacore/constants.pyDEFAULT_AGENT_TYPE,SUPPORTED_AGENT_TYPEScore/system/loader.pyagentsYAML data through toSystemConfigcore/models/__init__.pypipeline/evaluation/pipeline.pyconfig.api_enabledpipeline/evaluation/processor.pyconfig.api_enabledpipeline/evaluation/evaluator.pyconfig.api_enabledrunner/evaluation.pysystem_config.api_enabledtests/**/test_agents.pytests/**/test_system.pyapi_enabledproperty teststests/**/test_loader.pytests/**/test_data.pyTest plan
make pre-commitpasses (bandit, check-types, pyright, docstyle, ruff, pylint, black)Design Reference
🤖 Generated with Claude Code
Summary by CodeRabbit
New Features
Tests