fix(invariant): no_secret_echo only fails on string-valued suspect-named fields (#1713)#1714
Conversation
…med fields (#1713) The default context.no_secret_echo invariant flagged any response field whose name was in SUSPECT_PROPERTY_NAMES (authorization, api_key, apikey, bearer, x-api-key) regardless of the field's value. This over-rejected spec-legitimate structured fields — notably the authorization object on validation-result.json (structured authorization- validation payload, not a credential) and any seller-side extension fields named authorization under sync_accounts_response.accounts[] (additionalProperties: true). Symptom: BidMachine's sync_accounts failures in adcontextprotocol/adcp#4419 all surfaced as context.no_secret_echo. Diagnosis in #4419 pointed at Zod .strict() codegen lag, but verification of published SDK tarballs (5.25.1 / 6.12.0 / current) shows SyncAccountsResponseSchema.accounts[] already uses .passthrough(). Zod silently accepts the authorization field; the NAME dragnet in this invariant is the actual rejection. Fix: narrow findSecretEcho so the suspect-name check only fires when the field VALUE is a non-empty string. Structured object/array values on suspect-named fields pass through to the recursive walk, which still scans nested strings against BEARER_TOKEN_PATTERN and caller-supplied secret literals. The actual leak shapes the invariant was designed to catch remain caught. Five new test cases: - Suspect name with object value passes (the BidMachine case) - Suspect name with array value passes - Suspect name with empty string passes - Suspect name with non-empty string value still fails - Bearer literal nested inside structured suspect-named object still fails via the value-scan regression guard 113/113 invariant tests pass. Sequencing: - #1709 (merged via #1712) addresses Zod-reject misattribution (different path); this fix addresses the case where Zod ACCEPTS but the invariant over-rejects on field name. - #1707 (dynamic schema fetch + strict→passthrough) remains real architectural cleanup but does NOT unblock BidMachine. - #1711 (fgranata): the "Zod .strict() codegen lag" diagnosis was incorrect for the published SDK; this fix is the actual unblocker. Coordinated stance: #1685 (the SDK is a witness, not a translator). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
Following up on your guinea-pig invitation. Ran our prod against Empirically confirms your analysis. Two probes against
Three remaining failures we captured locally (each independent of #1714):
Shipping all three this afternoon. When grading retests against 7.1.0, would love a per-storyboard failure-list dump so we can diff against this local capture. Thanks again — the #1714 diagnostic trail made triangulation easy. |
…#1708) (#1715) Locks the post-7.1.0 attribution invariants so future refactors of comply()'s extractFailures can't silently reintroduce the BidMachine misattribution shape (adcp#4419). What's locked: 1. A storyboard step carrying both a synthesized response_schema failure (prepended by the runner per #1709 / PR #1712) and an assertion entry surfaces validation.check === 'response_schema' in ComplianceResult.failures — never 'assertion'. The attribution that was silently broken pre-7.1.0 (Zod rejects fell through to the next invariant, canonically context.no_secret_echo). 2. Skipped-invariant markers (passed: true entries the runner emits when short-circuiting invariants downstream of a schema failure per #1712) are correctly filtered out — only failed validations surface in `failures`. A future change that included passed: true entries would crowd out the real failure. 3. A clean BidMachine-shape response (structured authorization field passing no_secret_echo per #1713 / PR #1714) produces zero failures through the aggregation layer. 4. Multi-storyboard aggregation preserves per-storyboard (storyboard_id, step_id, validation.check) tuples. 5/6. Clean pass paths (no failures, skipped steps) produce empty failures. API change (minor): extractFailures (previously file-internal) is now exported from src/lib/testing/compliance/comply.ts so the regression test can call it directly with synthetic StoryboardResult fixtures. Functionally identical; just visibility. Scope correction relative to the original #1708 framing: the "cross-evaluator divergence" symptom was version-driven (different @adcp/sdk versions hitting #1713 and #1709 differently), not a true parity gap. Both root causes shipped in 7.1.0; this test is the durable guard for the aggregation-layer invariants those fixes depend on. 7/7 tests pass. Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Closes #1713.
The actual BidMachine root cause
Investigating #1707/#1711 surfaced something unexpected: BidMachine's
sync_accountscontext.no_secret_echofailures aren't caused by Zod.strict()codegen lag. I verified published @adcp/sdk tarballs at 5.25.1, 6.12.0, and current main — every one uses.passthrough()onSyncAccountsResponseSchema.accounts[]. Zod silently accepts theauthorizationfield.The rejection happens inside
context.no_secret_echo's name dragnet atsrc/lib/testing/storyboard/default-invariants.ts:184-190:The dragnet trips on field NAME alone (
authorization,api_key,apikey,bearer,x-api-key) regardless of value. BidMachine's response carriesauthorizationwith a structured object value (per the spec —compliance/cache/3.0.11/property/validation-result.jsondeclaresauthorizationas a structured authorization-validation payload, and theadditionalProperties: trueonsync_accounts_response.accounts[]permits seller extensions). The structured value isn't a credential leak; the dragnet false-positives.The fix
findSecretEchonow requires both a suspect name AND a non-empty string value:Object/array values on suspect-named fields pass through to the recursive walk, which still scans nested strings via
BEARER_TOKEN_PATTERNand caller-supplied secret literals. The actual leak shapes (bearer tokens at any depth, caller secret echoes, bare credential strings on suspect-named fields) all stay caught.Tests
Five new cases in
describe('default-invariants: context.no_secret_echo (widened whole-body scan)'):authorization) carrying an OBJECT value — the BidMachine caseapi_key) carrying an ARRAY value113/113 invariant tests pass.
Why this matters for the coordinated stance
response_schema)..strict()codegen lag" diagnosis was incorrect for the published SDK; this PR is the actual unblocker for the 27-point delta (63/128 comply vs 45/59 CLI). Will close once fgranata retests.So the order BidMachine cares about:
adcp_version#1707 — independent architectural cleanup; lower urgency now.Test plan
npm run buildcleannpm run format:checkcleannode --test test/lib/storyboard-default-invariants.test.js— 113/113 pass (5 new + all existing)sync_accountsfailures resolved.Coordinated stance: #1685 (the SDK is a witness, not a translator). Same anti-pattern as #1702 / #1705 / #1712: the SDK fabricated a credential-leak signal against a structured non-credential field the spec legally permits.
🤖 Generated with Claude Code