fix(compliance): rewrite deriveStoryboardStatuses for SDK 6.x scenario keys

[bokelley]

Summary

The compliance heartbeat has been writing zero rows to agent_storyboard_status since comply() switched to storyboard-driven testing. Every agent's dashboard storyboards_passing: 0/N was misleading — the parser was dropping the results.

Root cause

SDK 6.x emits one TestResult per phase of each storyboard, keyed <storyboard_id>/<phase_id> in result.tracks[].scenarios[].scenario (see @adcp/sdk compliance/storyboard-tracks.ts:54). The old deriveStoryboardStatuses walked the YAML's per-step comply_scenario field (bare names like signals_flow, capability_discovery) and looked them up in a Map keyed by the SDK's scenario strings. Every lookup missed → testedCount === 0 → every storyboard skipped at the continue guard. No rows written. No badges issued.

Scope

Direct DB query against prod:

agent_storyboard_status total rows: 6 (across 4 agents)
rows written by triggered_by='heartbeat': 0
existing rows: legacy bare-name keys from old manual runs (signals_baseline,
              capability_discovery, behavioral_analysis)

No heartbeat run has ever written to agent_storyboard_status. Affects every agent's badge eligibility, every specialism_status value, every storyboards_passing count.

Surfaced by escalation #329 — Evgeny's agent runs 30/30 scenarios clean but shows degraded because specialism_status.signal-owned = "untested" reads from a never-populated row.

Fix

Read SDK output directly. Group result.tracks[].scenarios[] by <storyboard_id> parsed from the scenario string, roll per-step pass counts up from each phase's steps array, fall back to phase-level counts when steps are absent. storyboardIds override is preserved for explicit-IDs callers (manual evals that need an untested entry when the runner didn't run a requested storyboard).

The YAML's comply_scenario field is no longer load-bearing for status mapping — the SDK already knows which storyboards it ran. Field is left in place (still useful for human documentation / planning).

Tests

server/tests/unit/derive-storyboard-statuses.test.ts — 9 cases:

• all-pass → status='passing', step counts roll up
• mixed pass/fail across phases → 'partial'
• all-fail → 'failing'
• phases without steps array → fall back to phase-level counts
• legacy bare-name scenarios → skipped
• empty input → []
• explicit storyboardIds with runner gap → 'untested' entry
• explicit storyboardIds ignoring extra storyboards in result
• multi-storyboard heartbeat result

All passing locally; type check clean.

Stack note

Orthogonal to Emma's #4247 compliance-state unification stack (#4250, #4263, #4264, #4268, #4274) which is about which tables carry compliance state (collapsing agent_test_history). This PR is about parsing the SDK output correctly into the existing tables. Different files (compliance-testing.ts vs her compliance-db.ts / member-tools.ts / registry-api.ts); rebases cleanly in either order.

Follow-ups

Once merged + deployed, the next heartbeat tick will start populating agent_storyboard_status properly for all 18 registered agents. A heartbeat sweep cycle drains in ~2h. After that, badge issuance via processAgentBadges should re-fire on the next heartbeat for any agent whose declared specialisms now have passing storyboard rows.

🤖 Generated with Claude Code

[bokelley]

Expert review pass — both clear

code-reviewer

No blockers. All actionable feedback addressed in c11abeb:

• JSDoc clarifies that steps_passed/steps_total are not cross-row comparable (some rows real step counts, some phase-fallback)
• Storyboard-id invariant (flat ids, no /) pinned in a comment
• result.tracks ?? [] defensive guard
• hasExplicitIds hoisted out of repeated check
• 3 new test cases: same-storyboard-across-tracks aggregation / result.tracks absent / non-string scenario value
• Changeset empty-frontmatter is intentional (matches repo pattern for non-version-bump changes)

12/12 vitest passing.

adtech-product-expert

Verdict: ship, with two product-side follow-ups (not blockers):

1. Vendor notice before deploy. 18 registered agents is hand-emailable. Frame as fidelity improvement, not regression — get ahead of the Evgeny-shape failure mode (vendor publicly claims "30/30 passing", dashboard contradicts them next heartbeat).
2. 30/30 vs storyboard-percentage dissonance is a real UX gap that this PR doesn't solve. Track summary counts steps executed; storyboard percentage counts steps passing inside a protocol bundle. Long-term: deprecate the track summary on the public dashboard, keep it in operator/debug view. Short-term: tooltip on the storyboard column. Follow-up issue, not a blocker.
3. Buy-side consumers: shape unchanged, values shift. Scope3 is the only known consumer at scale, and Wonderstruck (their agent) comes out better (10/10 on declared specialism). Low risk.
4. Badge issuance: "let the heartbeat run" — 2h drain is the staged rollout. Have someone watching first sweep.

Plan

• Merge once CI green
• Open follow-up issue for the dashboard reconciliation (track vs storyboard surface)
• Vendor outreach happens out-of-band before/after deploy per product team

Trust-but-verify quotes:

• sales_non_guaranteed for wonderstruck: 6 phases × 1 step in YAML = 6 declared; SDK runtime adds setup/seeding → 10 total → 10/10 passing. That's the specialism storyboard. The deeper protocol baseline (media_buy_seller, 89 steps) shows partial 42/89 — that's the real test depth.

[bokelley]

Got it — both reviewers clear, merge plan looks solid. Happy to open the follow-up issue for the track-summary vs. storyboard-percentage dashboard reconciliation if you want to hand that off; just say the word.

---

Generated by Claude Code