Ship StoryboardHarness — built-in compliance test harness for adopters via httpx.ASGITransport

[bokelley]

Summary

Every adopter of `adcp-client-python` who wants to run the compliance storyboard against their own seller in CI has to invent the same harness:

• Mount the full ASGI app via `build_app()` (or whatever entry point the seller exposes)
• Drive requests through `httpx.ASGITransport` to exercise the real wire envelope (MCP and A2A)
• Parse the response, extract `adcp_error` from `structuredContent` (MCP) or `Task.artifacts[0].parts[0].data` (A2A)
• Run scenarios from the bundled compliance cache against this harness
• Aggregate pass/fail

This is reinvented in every salesagent test file we've added today (see PR #330's `tests/integration/test_delegate_wire_envelope_cross_transport.py`). It's tedious and prone to drift across adopters.

Proposed surface

```python
from adcp.server.testing import StoryboardHarness

@pytest.fixture
def harness():
return StoryboardHarness(app=build_app(), seller_config=...)

def test_create_media_buy_compliance(harness):
result = harness.run_scenario(
"media_buy_seller/refine_products",
transport="mcp", # or "a2a", or "both"
)
assert result.passed, result.failures

def test_typed_error_translates(harness):
response = harness.invoke(
tool="update_media_buy",
payload={"media_buy_id": "does-not-exist", ...},
transport="a2a",
)
assert response.adcp_error.code == "MEDIA_BUY_NOT_FOUND"
```

The harness:

• Wraps the ASGI app and provides MCP/A2A transport adapters
• Reads compliance scenarios from the bundled cache
• Handles wire-shape envelope unpacking per transport
• Reports per-scenario pass/fail with the same JSON shape as `adcp storyboard run --json`

Why this matters

1. Adopters get compliance testing for free — no harness to maintain
2. Cross-transport contract tests become trivial — single fixture covers both transports
3. Spec drift detection — when the SDK bumps to a new spec version, every adopter's CI catches behavioral regressions automatically
4. Onboarding friction drops — "how do I test my seller against AdCP compliance?" has a one-line answer

Reference implementation

PR #330 in bokelley/salesagent (`tests/integration/test_delegate_wire_envelope_cross_transport.py`) is a working version for a single tool. Extending to scenario-level execution is the obvious next step.

Related

• adcp-client#1657/#1658 (alias cache reset — caught by careful manual inspection; would have been caught earlier by an integrated harness)
• adcp-client-python#652 (ValidationError → INVALID_REQUEST translation — same)
• bokelley/salesagent PR feat(decisioning): align AccountStore.resolution literal with JS + Tier 1 docs #330 (working cross-transport test pattern)

[bokelley]

Triage

Classification: Feature request
Bucket(s): handlers, testing
Status: drafting-pr
Milestone: omitted — no target version in issue text

What the experts said:

• dx-expert: place in adcp.testing (not a new adcp.server.testing module); name it SellerTestClient rather than StoryboardHarness to avoid implying scenario bundles that don't exist yet; ship invoke() only — no run_scenario() stub; typed AdcpErrorPayload dataclass return; from_platform() classmethod for DecisioningPlatform adopters
• code-reviewer: safe to execute with hard constraints — (a) adcp/testing/ only, (b) no run_scenario() anywhere, (c) MCP-only for v1 (transport="a2a" excluded — A2A has a separate ASGI app), (d) use in-process mcp.call_tool() to avoid SSE frame parsing, (e) passed is a computed property not a stored field, (f) adcp_error field is a dataclass/TypedDict not the exception class

My take: The core pain is real — build_test_client handles ASGI wiring but each adopter still has to write the JSON-RPC envelope construction + structuredContent["adcp_error"] extraction boilerplate by hand. SellerTestClient.invoke() eliminates exactly that. The run_scenario() half of the issue is blocked: there's no bundled scenario playbook cache in the Python SDK (the storyboard runner referenced in the issue is the external AAO compliance tool). Shipping invoke() now + noting the run_scenario() dependency in source is the right split.

Drafting PR for SellerTestClient.invoke() (MCP transport, in adcp.testing). run_scenario() and A2A transport deferred — this PR uses Refs #662, not Closes. A status comment will list what shipped vs. what remains.

---

Triaged by Claude Code. Session: https://claude.ai/code/session_01VZRaadSyoHDm7CU945e3Ba

---

Generated by Claude Code

[bokelley]

Triage status update — partial rollout via #666

Shipped in #666:

• SellerTestClient — in-process MCP harness; wraps DecisioningPlatform, calls mcp.call_tool(), returns ToolInvokeResult with .passed, .data, .adcp_error, .raw
• ToolInvokeResult — typed result container
• AdcpErrorPayload — typed adcp_error envelope fields (code, message, recovery, field, suggestion, retry_after, details)
• All three symbols exported from adcp.testing
• 18 new tests covering success path, error extraction, lazy init, instance reuse, and public import path

Not yet shipped (queued for follow-up):

• run_scenario() — blocked on bundled compliance scenario playbooks; the Python SDK currently has schema files in schemas/cache/3.0/ but no scenario playbooks, and no adcp storyboard run --json CLI equivalent. Tracked as Ship StoryboardHarness — built-in compliance test harness for adopters via httpx.ASGITransport #662.
• A2A transport support — A2A is served via a separate Starlette app (create_a2a_server()), not the same ASGI app. An in-process A2A test client needs its own primitive. Tracked as Ship StoryboardHarness — built-in compliance test harness for adopters via httpx.ASGITransport #662.

The class docstring in harness.py calls both out explicitly so adopters who read source aren't surprised.

---

Generated by Claude Code