Skip to content

Accept JSON string inputs in TaskNavigationEfficiencyEvaluator#46760

Draft
Copilot wants to merge 1 commit intomainfrom
copilot/redo-fix-in-sdk-repo-4999
Draft

Accept JSON string inputs in TaskNavigationEfficiencyEvaluator#46760
Copilot wants to merge 1 commit intomainfrom
copilot/redo-fix-in-sdk-repo-4999

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 6, 2026

Port of Azure/azureml-assets#4999 to the SDK repo. TaskNavigationEfficiencyEvaluator only accepted response/ground_truth as native Python lists/tuples, so callers passing serialized JSON strings (e.g. from data pipelines) failed validation.

Changes

  • _evaluators/_task_navigation_efficiency/_task_navigation_efficiency.py: At the top of _do_eval, if response or ground_truth is a str, attempt json.loads and replace the value on success.
  • _evaluators/_common/_validators/_task_navigation_efficiency_validator.py: Same fallback at the top of validate_eval_input (the SDK uses a separate validator class, not an inline method like the upstream evaluator).
  • tests/unittests/test_task_navigation_efficiency_evaluators.py: Added test_json_stringified_valid_inputs covering the EXACT_MATCH path with JSON-stringified inputs.
  • CHANGELOG.md: Bug fix entry under 1.16.7 (Unreleased).

On parse failure the original string is left in place so downstream validation surfaces the error as before.

for key in ("response", "ground_truth"):
    value = eval_input.get(key)
    if isinstance(value, str):
        try:
            eval_input[key] = json.loads(value)
        except (ValueError, TypeError):
            pass

Note: the SDK uses response/ground_truth parameter names rather than actions/expected_actions from azureml-assets; keys were adapted accordingly. No spec.yaml version bump applies here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants