-
Notifications
You must be signed in to change notification settings - Fork 2.8k
Description
Describe the bug
The new implementation in the ToolTrajectoryCriterion class doesn't accept string values for match_type (e.g., "ANY_ORDER") causing a Pydantic validation error. The enum expects integer values (0, 1, or 2) but in the documentation shows string values. It works with integers.
To Reproduce
Create an evaluation configuration with a string value for match_type:
from google.adk.evaluation.eval_metrics import EvalMetric, ToolTrajectoryCriterion, PrebuiltMetrics
eval_metric = EvalMetric(
threshold=0.5,
metric_name=PrebuiltMetrics.TOOL_TRAJECTORY_AVG_SCORE.value,
criterion={
"threshold": 0.1,
"match_type": "ANY_ORDER" # Using string as documented
}
)Error:
pydantic_core._pydantic_core.ValidationError: 1 validation error for ToolTrajectoryCriterion
match_type
Input should be 0, 1 or 2 [type=enum, input_value='ANY_ORDER', input_type=str]
For further information visit https://errors.pydantic.dev/2.12/v/enum
Expected behavior
The criterion should accept string values ("EXACT", "IN_ORDER", "ANY_ORDER") as shown in the ADK documentation.
Desktop:
- OS: macOS
- Python version: 3.13.0
- ADK version: 1.19.0
Root Cause
The MatchType enum in src/google/adk/evaluation/eval_metrics.py uses integer values (0, 1, 2), but Pydantic v2
doesn't automatically convert string enum names to enum values without an explicit validator.
Proposed Solution
Add a Pydantic field_validator to the ToolTrajectoryCriterion.match_type field that converts string enum names to their corresponding enum values if we want to keep compatibility with integer values.
I've already tested a fix locally and can open a PR if that would be helpful, with comprehensive test coverage of course. Let me know if you'd like me to contribute the fix.
Thanks for all hard work!