docs: define learning runtime verification matrix

[devkade]

Summary

• Adds docs/learning-runtime-verification-matrix.md as the Design: define verification matrix for learning parallel runtime #191 verification contract for the learning parallel runtime.
• Covers unit, integration, E2E smoke, and failure-mode matrices across schema/events/DAG/claims/workers/policy/reward/integration/retention/storage.
• Adds doc contract tests and README linkage so the matrix remains discoverable and tied to child-issue closeout evidence.

Linked issue

Closes #191

Problem

#167 now has runtime MVP slices through task graph, claim/lease, and worker execution. Before runtime readiness/default claims, #191 needs a concrete verification matrix that maps every runtime invariant to test categories, smoke artifacts, failure-mode expectations, fixtures, and child-issue closeout evidence.

Without that matrix, later runtime PRs can claim readiness from partial unit coverage or narrative worker output instead of executable evidence.

Options considered

1. Add only a short README checklist.
   • Pros: smallest diff.
   • Cons: too thin for Design: define verification matrix for learning parallel runtime #191 acceptance criteria; hard to map child issues and failure modes.
2. Add a dedicated design doc plus contract tests.
   • Pros: satisfies Design: define verification matrix for learning parallel runtime #191 directly, keeps docs discoverable, and protects the key headings/coverage with tests.
   • Cons: adds one new docs file and one test file.
3. Implement CI/runtime probes now.
   • Pros: stronger executable readiness.
   • Cons: out of scope; Design: define verification matrix for learning parallel runtime #191 explicitly says no implementation and no CI policy change unless split.

Selected approach

Selected option: dedicated design doc plus contract tests.

Why: it satisfies the design-only issue without changing runtime behavior, while still making the matrix enforceable enough that future changes cannot silently drop the verification contract.

Implementation by file/surface

• docs/learning-runtime-verification-matrix.md
  • Defines verification layers: unit, integration, E2E smoke, failure mode.
  • Maps runtime areas to required coverage and current/target test surfaces.
  • Defines minimum E2E evidence bundle: state.json, events.jsonl, worker report fixtures, objective/evaluation, reward ledger, integration dry-run, command output.
  • Defines child-issue closeout evidence for Design: define RunState schemas, versions, and migration policy #185, Design: define runtime event taxonomy, replay, and recovery semantics #186, Design: define portable agent adapter and execution substrate contracts #188, Sub-roadmap: DAG runtime phase 1 — task graph and readiness #194, Sub-roadmap: DAG runtime phase 2 — claim, lease, and stale ownership #197, Sub-roadmap: DAG runtime phase 3 — worker execution, heartbeat, and evidence-gated completion #196, Design: define verification matrix for learning parallel runtime #191, Design: define reward calculation, penalties, PolicyHint, and calibration #189, Design: define policy simulator features, estimators, and exploration safety #187, Design: define integration, repair, and supersession semantics #190, and Sub-roadmap: DAG runtime phase 4 — IntegrationCandidate and repair loop #195.
• test/learning-runtime-verification-matrix.test.ts
  • Verifies the matrix includes unit/integration/E2E/failure/closeout sections.
  • Verifies required runtime areas and child issue gates remain present.
  • Verifies README links the doc.
• README.md
  • Adds the matrix to the documented layout.

Why this fixes it

The new matrix directly addresses every #191 acceptance criterion:

• unit test matrix defined;
• integration test matrix defined;
• E2E smoke path defined;
• failure-mode tests defined;
• required fixtures/artifacts listed;
• minimum evidence for child issue closeout documented.

It also keeps runtime-readiness claims blocked until executable or recorded smoke evidence covers the MVP rows.

QA / Verification

• npm ci — pass.
• npm run check — pass.
• npm run check:unused — pass.
• npm test -- test/learning-runtime-verification-matrix.test.ts — pass; package script ran full suite (537 tests, 526 pass, 11 skipped).
• npm run quality:budgets — pass with existing non-failing code_smells=60 warning.
• git diff --cached --check — pass.
• Review-size check — 3 files changed, 167 insertions(+).

Anomalies observed

• Fresh worktree needed npm ci; initial npm run check failed because @types/node was not installed yet.
• npm test -- test/learning-runtime-verification-matrix.test.ts runs the full suite because the package script includes test/*.test.ts.

Risks / Follow-up

• This is a design/test-contract slice only. It does not add runtime probes or CI policy changes.
• Roadmap: objective-driven learning parallel runtime harness #167 still needs roadmap reconciliation after Design: define verification matrix for learning parallel runtime #191 closes, especially to mark Design: define verification matrix for learning parallel runtime #191 complete and move next work toward Design: define reward calculation, penalties, PolicyHint, and calibration #189/Design: define policy simulator features, estimators, and exploration safety #187 shallow learning records or Design: define integration, repair, and supersession semantics #190/Sub-roadmap: DAG runtime phase 4 — IntegrationCandidate and repair loop #195 post-MVP integration/repair.

kapi-agent review expectations and current-head merge gate

• Expected review focus: design completeness against Design: define verification matrix for learning parallel runtime #191 acceptance criteria, doc discoverability, and no accidental runtime behavior change.
• Merge gate: require current-head kapi-agent approval and successful kapi-agent/review / formal approval checks before merge.

[devkade]

@kapi-agent review

Revision explanation for current head 263d021149865b5b4bd73183d5ccf453c62f5303:

What changed:

• Added docs/learning-runtime-verification-matrix.md for Design: define verification matrix for learning parallel runtime #191.
• Added test/learning-runtime-verification-matrix.test.ts to assert the matrix keeps unit/integration/E2E/failure/closeout sections, required runtime areas, child-issue closeout gates, and README discoverability.
• Updated README.md layout to link the new matrix.

Why this closes #191:

• The document defines the required unit test matrix, integration test matrix, E2E smoke path, failure-mode matrix, fixtures/artifacts, and child-issue closeout evidence.
• It stays design-only: no runtime behavior, CI policy, storage behavior, or adapter behavior changed.
• It explicitly blocks runtime-readiness/default claims until executable or recorded smoke evidence covers the MVP rows.

Verification:

• npm ci passed.
• npm run check passed.
• npm run check:unused passed.
• npm test -- test/learning-runtime-verification-matrix.test.ts passed; package script ran full suite (537 tests, 526 pass, 11 skipped).
• npm run quality:budgets passed with existing non-failing code_smells=60 warning.
• git diff --cached --check passed before commit.
• Review-size check: 3 files changed, 167 insertions(+).

[sentinel-agent-bot]

kapi-agent review completed

The formal GitHub PR review and required check were submitted separately. This request comment is kept concise to avoid duplicating the full review body.

kapi-agent request lifecycle:

• Status: COMPLETED
• Head: 263d02114986
• Formal review event: APPROVE
• Review API status: 200
• Required check API status: 201

[sentinel-agent-bot]

kapi-agent review

Verdict: APPROVE

Final approval summary

Review journey

This PR’s original goal is to close #191 by defining the verification contract for the learning parallel runtime before any runtime-readiness/default claims are made. The semantic scope is bounded to documentation plus doc contract tests: one new verification matrix, one README discoverability link, and one test file that guards key matrix sections and issue closeout gates. There was no prior kapi-agent review on this head, so no review/fix ping-pong was required.

What changed

• Added docs/learning-runtime-verification-matrix.md.
• Added test/learning-runtime-verification-matrix.test.ts.
• Updated README.md to link the new matrix from the documented layout.

Why this is correct

The matrix covers the expected #191 surfaces: unit, integration, E2E smoke, failure modes, required artifacts/fixtures, and child-issue closeout evidence. It also explicitly keeps the change design-only and blocks runtime-readiness claims until executable coverage or recorded smoke evidence exists for MVP-critical rows.

Evidence

• Verifier gate: PASS.
• Size gate: PASS, 167 changed lines under the 200-line semantic review threshold.
• Revision-explanation status: not required, but provided.
• Ilchul review harness: PASS, LOW_RISK, no blocking findings.
• Verified evidence includes npm ci && npm run verify passing.
• Inspected files: README.md, docs/learning-runtime-verification-matrix.md, and test/learning-runtime-verification-matrix.test.ts.

Remaining risks and approval rationale

Remaining risk is limited to the matrix being a design contract rather than executable runtime validation. That is acceptable for #191 because the PR clearly states this is not a runtime or CI behavior change, and the tests preserve discoverability and minimum contract structure. Approval is justified because all gates pass and no blocking correctness, safety, workflow-contract, or regression issue remains.

Blocking issues

None.

Warnings / risks

• docs/learning-runtime-verification-matrix.md: the document intentionally names some “target” test surfaces that may not exist or may not fully cover the stated behavior yet. This is acceptable for a design-only matrix, but future runtime-readiness PRs should not treat these rows as satisfied without executable evidence.

Suggestions

• Consider adding a trailing newline to docs/learning-runtime-verification-matrix.md in a follow-up cleanup.
• Future implementation PRs should link their new tests/smoke artifacts back to the exact matrix rows they satisfy.

Looks good

• The matrix clearly separates unit, integration, E2E smoke, and failure-mode expectations.
• The child-issue closeout table gives future PRs a concrete evidence contract.
• README discoverability is covered.
• The contract tests are appropriately lightweight for a docs/design slice and guard against accidental removal of the important sections and issue gates.

Verification notes

• Verifier gate status: PASS — npm ci && npm run verify exited 0.
• Size gate status: PASS — 167 changed lines, below the 200-line review threshold.
• Revision-explanation status: not required; explanation found.
• Ilchul review harness gate: PASS — LOW_RISK, no blocking findings.
• Local/CI evidence in the PR body and gate output includes tests, type/check gates, unused checks, and quality budgets passing.

---

Engine: pi