Skip to content

Instrument requirement validators to track validation attempts and failures #473

@psschwei

Description

@psschwei

Description:
Instrument requirement validators to track validation attempts and failures.

Detailed Requirements:

  1. Add spans for validation:
    • validator.check - Individual validation check
  2. Add attributes:
    • requirement_type - Type of requirement (python, markdown, llmaj, etc.)
    • requirement_name - Specific requirement identifier
    • status - pass/fail
    • failure_reason - Reason for failure (on fail)
  3. Add span events for:
    • Validation start with input summary
    • Validation result with details
  4. Instrument all requirement types:
    • PythonRequirement
    • MarkdownRequirement
    • ToolCallRequirement
    • ALoraRequirement
    • LLMaJRequirement

Files to Modify:

  • mellea/stdlib/requirements/base.py - Base instrumentation
  • mellea/stdlib/requirements/python.py - Python requirement
  • mellea/stdlib/requirements/markdown.py - Markdown requirement
  • mellea/stdlib/requirements/tool_call.py - Tool call requirement
  • mellea/stdlib/requirements/alora.py - ALoRA requirement
  • mellea/stdlib/requirements/llmaj.py - LLMaJ requirement

Acceptance Criteria:

  • Each validation check has a span
  • Pass/fail status captured
  • Failure reasons recorded
  • All requirement types instrumented

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions