Status code

[thorrester]

Pull Request

Short Summary

• Odds and ends for opsml compatibility
• rename field_path to context_path -> better naming consistency with GenAIEvalRecord

Evaluate array elements

Add the ability to validate the elements of an array with assertion tasks
Use Case:

• Agent may return list of objects and user may want to ensure all object meet certain criteria
• add optional item_context_path field to AssertionTask

Example

result = [{"foo": 10}, {"foo":20}]

context = {
    "result": result
}

task = AssertionTask(
         id="Check Foo,
         context_path="result",
         item_context_path="foo"
         operator=ComparisonOperator.GREATER_THAN,
         expected_value=9,
         description="Check foo greater than 9"
    )

Here context_path refers to "result" key in the context of the GenAIEvalRecord and item_context_path refers to the key foo of each element in the array. If item_context_path is set, and the extracted context is an array, each element will be validated.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 65.06%. Comparing base (11b7d26) to head (df91113).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #209   +/-   ##
=======================================
  Coverage   65.06%   65.06%           
=======================================
  Files          18       18           
  Lines         249      249           
=======================================
  Hits          162      162           
  Misses         87       87           

Files with missing lines	Coverage Δ
py-scouter/python/scouter/evaluate/__init__.py	100.00% <ø> (ø)

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.