fix(conformance): grade storyboards not_applicable when required_tools are missing#1682
Merged
Merged
Conversation
…s missing Before running each storyboard, check required_tools against the agent's discovered toolset. Storyboards with missing tools are pushed to notApplicable and skipped rather than run — preventing cascading step-level skips from pulling the track to partial. Also corrects storyboards_executed, groupByTrack, and extractFailures to reference runnableStoryboards rather than the full expanded set. Closes #1680 https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX
… path; add storyboards_not_applicable - Wrap the required_tools filter in else-branch so explicit storyboard IDs (options.storyboards) bypass it — they are an operator override - Add storyboards_not_applicable field to ComplianceResult so AAO consumers can distinguish intentional exclusions from missing coverage https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX
…d type Storyboard.title is string (non-nullable) in storyboard/types.ts; drop the sb.index_title fallback which doesn't exist on the type. https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX
This was referenced May 11, 2026
Closed
Contributor
Author
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1680
Summary
comply()was running every capability-resolved storyboard even when the agent's discovered toolset was missing one or more tools declared in the storyboard'srequired_toolsfield. The cascading step-level skips caused the storyboard to gradepartial, which propagated to the track as a false failure — visible in the Wonderstruck probe (59/66 partialinstead of59/59 + 7 not-applicable).Changes:
complyImpl, afterexpandScenarios, buildrunnableStoryboardsby filtering out storyboards/scenarios whoserequired_toolsaren't in the agent's discovered toolset. Filtered entries are pushed tonotApplicablewithreason: "missing required_tools: <list>"and get a syntheticnot_applicableresult (consistent with the existing version-gating path).options.storyboards) bypass the filter — they are an operator override and run regardless ofrequired_tools.groupByTrack,poolTrackSet,extractFailures, andstoryboards_executedto referencerunnableStoryboardsrather than the full expanded set.storyboards_not_applicable?: string[]toComplianceResultso AAO consumers can distinguish intentional exclusions from missing coverage without parsing per-track results.What was tested
npm run format:check✅npx tsc— only pre-existing env errors (missing@types/node, deprecatedmoduleResolution=node10); no errors in changed files. node_modules not installed in this environment — full test suite runs in CI (same constraint as fix(cli): propagate --allow-http to MCPOAuthProvider.validateResourceURL #1671).Storyboard.title: string(non-nullable,storyboard/types.ts:18);required_tools?: string[](types.ts:33);NotApplicableStoryboard.storyboard_title: string(storyboard/compliance.ts:118) — all types align.Pre-PR review
storyboards_not_applicablefield added;sb.titleis non-nullable so no fallback needed; all fiveapplicableStoryboardsreferences correctly updated torunnableStoryboardsnot_applicablewithoverall_passed: trueis spec-correct per adcp#4226/#4228; AAO track rollup is clean; explicit-storyboard bypass semantics documented in comment;storyboards_not_applicablesatisfies AAO grader contract. Note:runWithDegradedProfiledoesn't emitstoryboards_not_applicable(vacuously harmless — degraded profile produces no capability-driven exclusions) — follow-up before grader contract is finalized.Nits noted (not fixed in this PR)
node_modulesnot available in this environment; test coverage forcomply.tsis a known gaprunWithDegradedProfiledoesn't emitstoryboards_not_applicable— currently harmless (degradednotApplicableis always empty) but worth adding before the AAO grader reads this fieldSession: https://claude.ai/code/session_019y9VpBe6T9oSU69GzRicDX
Generated by Claude Code