Skip to content

tool_trajectory_avg_score class doesn't accept string values for match_type #3711

@alexhermida

Description

@alexhermida

Describe the bug
The new implementation in the ToolTrajectoryCriterion class doesn't accept string values for match_type (e.g., "ANY_ORDER") causing a Pydantic validation error. The enum expects integer values (0, 1, or 2) but in the documentation shows string values. It works with integers.

To Reproduce
Create an evaluation configuration with a string value for match_type:

from google.adk.evaluation.eval_metrics import EvalMetric, ToolTrajectoryCriterion, PrebuiltMetrics

eval_metric = EvalMetric(
    threshold=0.5,
    metric_name=PrebuiltMetrics.TOOL_TRAJECTORY_AVG_SCORE.value,
    criterion={
        "threshold": 0.1,
        "match_type": "ANY_ORDER"  # Using string as documented
    }
)

Error:

  pydantic_core._pydantic_core.ValidationError: 1 validation error for ToolTrajectoryCriterion
  match_type
    Input should be 0, 1 or 2 [type=enum, input_value='ANY_ORDER', input_type=str]
      For further information visit https://errors.pydantic.dev/2.12/v/enum

Expected behavior
The criterion should accept string values ("EXACT", "IN_ORDER", "ANY_ORDER") as shown in the ADK documentation.

Desktop:

  • OS: macOS
  • Python version: 3.13.0
  • ADK version: 1.19.0

Root Cause
The MatchType enum in src/google/adk/evaluation/eval_metrics.py uses integer values (0, 1, 2), but Pydantic v2
doesn't automatically convert string enum names to enum values without an explicit validator.

Proposed Solution
Add a Pydantic field_validator to the ToolTrajectoryCriterion.match_type field that converts string enum names to their corresponding enum values if we want to keep compatibility with integer values.

I've already tested a fix locally and can open a PR if that would be helpful, with comprehensive test coverage of course. Let me know if you'd like me to contribute the fix.

Thanks for all hard work!

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluationneeds-review[Status] The PR is awaiting review from the maintainer

Type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions