Skip to content

fix(conformance): grade storyboards not_applicable when required_tools are missing#1682

Merged
bokelley merged 5 commits into
mainfrom
claude/issue-1680-required-tools-not-applicable
May 11, 2026
Merged

fix(conformance): grade storyboards not_applicable when required_tools are missing#1682
bokelley merged 5 commits into
mainfrom
claude/issue-1680-required-tools-not-applicable

Conversation

@bokelley
Copy link
Copy Markdown
Contributor

Closes #1680

Summary

comply() was running every capability-resolved storyboard even when the agent's discovered toolset was missing one or more tools declared in the storyboard's required_tools field. The cascading step-level skips caused the storyboard to grade partial, which propagated to the track as a false failure — visible in the Wonderstruck probe (59/66 partial instead of 59/59 + 7 not-applicable).

Changes:

  • In complyImpl, after expandScenarios, build runnableStoryboards by filtering out storyboards/scenarios whose required_tools aren't in the agent's discovered toolset. Filtered entries are pushed to notApplicable with reason: "missing required_tools: <list>" and get a synthetic not_applicable result (consistent with the existing version-gating path).
  • Explicit storyboard IDs (options.storyboards) bypass the filter — they are an operator override and run regardless of required_tools.
  • Corrects groupByTrack, poolTrackSet, extractFailures, and storyboards_executed to reference runnableStoryboards rather than the full expanded set.
  • Adds storyboards_not_applicable?: string[] to ComplianceResult so AAO consumers can distinguish intentional exclusions from missing coverage without parsing per-track results.

What was tested

  • npm run format:check
  • npx tsc — only pre-existing env errors (missing @types/node, deprecated moduleResolution=node10); no errors in changed files. node_modules not installed in this environment — full test suite runs in CI (same constraint as fix(cli): propagate --allow-http to MCPOAuthProvider.validateResourceURL #1671).
  • Manual type verification: Storyboard.title: string (non-nullable, storyboard/types.ts:18); required_tools?: string[] (types.ts:33); NotApplicableStoryboard.storyboard_title: string (storyboard/compliance.ts:118) — all types align.

Pre-PR review

  • code-reviewer: approved — explicit-storyboard bypass correctly guarded; storyboards_not_applicable field added; sb.title is non-nullable so no fallback needed; all five applicableStoryboards references correctly updated to runnableStoryboards
  • ad-tech-protocol-expert: approved — not_applicable with overall_passed: true is spec-correct per adcp#4226/#4228; AAO track rollup is clean; explicit-storyboard bypass semantics documented in comment; storyboards_not_applicable satisfies AAO grader contract. Note: runWithDegradedProfile doesn't emit storyboards_not_applicable (vacuously harmless — degraded profile produces no capability-driven exclusions) — follow-up before grader contract is finalized.

Nits noted (not fixed in this PR)

  • No unit test added for the filter path — node_modules not available in this environment; test coverage for comply.ts is a known gap
  • runWithDegradedProfile doesn't emit storyboards_not_applicable — currently harmless (degraded notApplicable is always empty) but worth adding before the AAO grader reads this field

Triage-managed PR. This bot does not currently iterate on
review comments or PR conversation threads (only on the source
issue). To unblock:

  • Push fixup commits directly: gh pr checkout 1681
    fix → push.
  • Or re-trigger: comment /triage execute on the source
    issue.

See adcp#3121
for context.

Session: https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX


Generated by Claude Code

claude added 3 commits May 11, 2026 13:40
…s missing

Before running each storyboard, check required_tools against the
agent's discovered toolset. Storyboards with missing tools are pushed
to notApplicable and skipped rather than run — preventing cascading
step-level skips from pulling the track to partial.

Also corrects storyboards_executed, groupByTrack, and extractFailures
to reference runnableStoryboards rather than the full expanded set.

Closes #1680

https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX
… path; add storyboards_not_applicable

- Wrap the required_tools filter in else-branch so explicit storyboard
  IDs (options.storyboards) bypass it — they are an operator override
- Add storyboards_not_applicable field to ComplianceResult so AAO
  consumers can distinguish intentional exclusions from missing coverage

https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX
…d type

Storyboard.title is string (non-nullable) in storyboard/types.ts;
drop the sb.index_title fallback which doesn't exist on the type.

https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX
@bokelley
Copy link
Copy Markdown
Contributor Author

Tracking under #1685 — hat 5 of 5 (partial grade inflation). Do not merge before #1684 (the visibility keystone). #1690 builds on this PR's not_applicable plumbing for the webhook_receiver requirement; sequence is #1684#1683/#1681 → this → #1690.

@bokelley
Copy link
Copy Markdown
Contributor Author

Noted — merge sequence recorded: #1684#1683/#1681 → this PR → #1690. This PR stays draft until #1684 lands. Tracking parent: #1685 (hat 5/5).


Generated by Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

comply() grades storyboards 'partial' when required_tools are missing — should grade not_applicable

2 participants