feat(text-metrics): split qa_accuracy

[davidberenstein1957]

Summary

• Split qa_accuracy into its own stacked branch/PR
• Add QAAccuracyMetric implementation
• Wire GenEval benchmark to qa_accuracy + clip_score

Test plan

• Run uv run pytest tests/evaluation/test_text_metrics.py -k qa_accuracy

Made with Cursor

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Comment @cursor review or bugbot run to trigger another review on this PR

^{Reviewed by Cursor Bugbot for commit 15db155. Configure here.}

[cursor]

GenEval benchmark uses lenient mean aggregation

Medium Severity

The GenEval benchmark's qa_accuracy metric is intended for "strict QA" (all-or-nothing aggregation). However, Task.from_benchmark doesn't pass the all_or_nothing kwarg, causing QAAccuracyMetric to default to mean aggregation. This leads to inflated scores that are not comparable to the paper's reference.

 

^{Reviewed by Cursor Bugbot for commit 15db155. Configure here.}

[cursor]

Redundant metric_units self-assignment in __init__

Low Severity

Setting self.metric_units = type(self).metric_units is a no-op because metric_units is already declared as a class attribute and nothing earlier in __init__ (neither super().__init__ nor the surrounding code) overrides it. The line just shadows the class attribute with the same value and adds maintenance noise.

 

^{Reviewed by Cursor Bugbot for commit 15db155. Configure here.}