Skip to content

feat(pipeline): add wave-audit pipeline for implementation fidelity auditing#308

Merged
nextlevelshit merged 1 commit intomainfrom
305-audit-pipeline
Mar 12, 2026
Merged

feat(pipeline): add wave-audit pipeline for implementation fidelity auditing#308
nextlevelshit merged 1 commit intomainfrom
305-audit-pipeline

Conversation

@nextlevelshit
Copy link
Copy Markdown
Collaborator

@nextlevelshit nextlevelshit commented Mar 11, 2026

Summary

  • Add wave-audit pipeline definition (.wave/pipelines/wave-audit.yaml) with 4 steps: collect-inventory, audit-items, compose-triage, and publish
  • Add JSON Schema contracts for all step outputs: audit-inventory, audit-findings, audit-triage-report, and audit-publish-result
  • Pipeline uses gh CLI to fetch closed issues/merged PRs, performs static analysis verification against HEAD, produces a prioritized triage report
  • Supports scoped audits (time range, label filters), resume via --from-step, and optional GitHub issue creation for actionable findings
  • All analysis steps use read-only personas (navigator/auditor); only the optional publish step uses craftsman

Spec

See specs/305-audit-pipeline/spec.md for the full feature specification including user stories, requirements, and success criteria.

Test Plan

  • go test -race ./... passes with all tests green
  • Pipeline YAML loads and validates via existing manifest test suite
  • Contract schemas are valid JSON Schema draft-07
  • Pipeline structure follows established patterns (doc-audit, wave-evolve)
  • Manual: wave run wave-audit on a repository with closed issues to verify end-to-end flow

Known Limitations

  • Audit accuracy depends on LLM persona quality — static analysis cannot verify runtime behavior
  • Large repositories (500+ issues) may approach adapter context limits during the audit step
  • Scope parsing relies on persona interpretation of natural-language input, not a structured parser

Closes #305

…uditing

Add a 4-step pipeline that audits closed issues and merged PRs against
the current codebase to detect implementation gaps, regressions, and
incomplete features.

Steps:
- collect-inventory: fetches closed issues/merged PRs via gh CLI
- audit-items: static analysis verification with fidelity classification
- compose-triage: aggregates findings into prioritized triage report
- publish: creates GitHub issues for actionable findings

Supports scoped audits (time range, label filters), resume via
--from-step, and JSON schema contract validation on all step outputs.

Closes #305
@nextlevelshit
Copy link
Copy Markdown
Collaborator Author

Code Review (Wave Pipeline)

Verdict: REQUEST_CHANGES

Solid pipeline design with clean DAG decomposition and good use of contract validation at every handover. Two issues should be fixed before merge; the rest are recommendations.


Critical Issues

1. Missing must_pass: true on publish step contract.wave/pipelines/wave-audit.yaml:340-348

The publish step's handover contract defaults to soft failure. If the persona writes malformed JSON, the pipeline reports success with corrupt output. The analogous doc-audit.yaml publish step sets must_pass: true. This should match.

2. summary not required in inventory schema.wave/contracts/audit-inventory.schema.json

The pipeline prompt and data model both treat summary (with total_issues, total_prs, excluded_not_planned) as required output, but the schema only requires ["scope", "items", "timestamp"]. Downstream steps depend on these counts. Add "summary" to the top-level required array.


Suggested Improvements

Schema tightening (5 items, all straightforward):

  • Add "by_category" to required fields in findings summary (.wave/contracts/audit-findings.schema.json:46-48) — compose-triage depends on these counts
  • Add "minItems": 1 to evidence arrays in findings schema — empty evidence violates FR-009
  • Add "format": "uri" to all url/item_url fields across the 4 schemas — timestamps already use format validation, URLs should too
  • Add "required": ["code", "message"] to the error object in audit-publish-result.schema.json
  • Consider additionalProperties: false on schema objects, or document the intentional choice to allow extra fields

Dependency hygiene.wave/pipelines/wave-audit.yaml:160-170

compose-triage injects an artifact from collect-inventory but only declares audit-items as a dependency. This works via transitive ordering but is fragile and unlike any other pipeline in the codebase. Add collect-inventory as an explicit dependency.

Scalability.wave/pipelines/wave-audit.yaml:98-156

The audit-items step processes all inventory items in a single adapter session. For the SC-001 target (500 issues + 300 PRs), this will likely exceed context limits. Consider adding a max_items cap or documenting the expected scaling boundary.

Testing — No new tests cover schema validity or DAG resolution for this pipeline. A test that loads the YAML and validates each schema against example data would prevent regressions.

Process — The three spec checklists (contracts.md, pipeline-design.md, review.md) are entirely unchecked. Complete them as part of this PR.


Positive Observations

  • Contract validation at every handover: All 4 steps enforce JSON Schema contracts with retry semantics — data integrity is well-protected across the pipeline
  • Read-only analysis steps: Steps 1-3 use read-only personas (github-analyst, navigator). Only the final publish step can create GitHub issues, minimizing blast radius
  • Input sanitization: User input is sanitized before template expansion into prompts, mitigating prompt injection via {{ input }}
  • Deny-first permissions: The github-analyst persona explicitly denies gh issue edit/create/close, preventing the inventory step from modifying GitHub state
  • Worktree isolation: All steps run in ephemeral worktrees — no writes reach the source repository
  • All existing tests pass: 30-31 packages, 0 failures across go test ./...
  • Clean DAG decomposition: The 4-step pipeline (collect → audit → triage → publish) has clear separation of concerns with well-defined artifact boundaries

Generated by Wave gh-pr-review pipeline

@nextlevelshit nextlevelshit merged commit 4072274 into main Mar 12, 2026
6 checks passed
@nextlevelshit nextlevelshit deleted the 305-audit-pipeline branch March 12, 2026 18:30
nextlevelshit added a commit that referenced this pull request Apr 12, 2026
feat(pipeline): add wave-audit pipeline for implementation fidelity auditing
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(pipeline): add closed-issue/PR audit pipeline for implementation gap detection

1 participant