Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# @copilotkit/aimock

## [Unreleased]

### Added

- **Snapshot-style recording** — When `X-Test-Id` is present, recorded fixtures are saved to `<fixturePath>/<slugified-testId>/<provider>.json` instead of timestamp-based filenames. Multiple fixtures for the same test+provider merge into one file. Stable paths enable meaningful PR diffs and easy test-to-fixture mapping. (Feature request by @jantimon, issue #155)

## [1.18.0] - 2026-05-04

### Added
Expand Down
36 changes: 36 additions & 0 deletions docs/examples/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -368,6 +368,42 @@ <h3>Record &amp; Replay</h3>
}</code></pre>
</div>

<h3>Snapshot Recording (per-test fixtures)</h3>
<p>
Send <code>X-Test-Id</code> from your test runner to organize recorded fixtures into
per-test directories. Here is a Playwright example:
</p>
<div class="code-block">
<div class="code-block-header">playwright/setup.ts <span class="lang-tag">ts</span></div>
<pre><code><span class="kw">import</span> { test } <span class="kw">from</span> <span class="str">"@playwright/test"</span>;

test.<span class="fn">beforeEach</span>(<span class="kw">async</span> ({ <span class="op">page</span> }, <span class="op">testInfo</span>) <span class="kw">=&gt;</span> {
<span class="cm">// Route all LLM traffic through aimock with a test ID header</span>
<span class="kw">await</span> <span class="op">page</span>.<span class="fn">setExtraHTTPHeaders</span>({
<span class="str">"X-Test-Id"</span>: <span class="op">testInfo</span>.<span class="prop">title</span>,
});
});</code></pre>
</div>
<p>
The resulting fixture directory layout groups each test&rsquo;s recordings by provider:
</p>
<div class="code-block">
<div class="code-block-header">
Recorded fixture tree <span class="lang-tag">text</span>
</div>
<pre><code>fixtures/recorded/
should-greet-the-user/
openai.json
anthropic.json
should-handle-tool-calls/
openai.json</code></pre>
</div>
<p>
See
<a href="/record-replay#snapshot-style-recording">Snapshot-Style Recording</a> for the
full workflow, including replay and drift detection.
</p>

<!-- ─── Full Suite ─────────────────────────────────────────── -->

<h2>Full Suite</h2>
Expand Down
10 changes: 10 additions & 0 deletions docs/fixtures/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -493,6 +493,16 @@ <h3>From a directory</h3>
<span class="op">mock</span>.<span class="fn">loadFixtureDir</span>(<span class="str">"./fixtures"</span>);</code></pre>
</div>

<div class="info-box">
<p>
<strong>Snapshot-style recording:</strong> When recording with <code>X-Test-Id</code>,
fixtures are automatically organized into per-test directories
(<code>&lt;fixturePath&gt;/&lt;test-slug&gt;/&lt;provider&gt;.json</code>). See
<a href="/record-replay#snapshot-style-recording">Snapshot-Style Recording</a> for
details.
</p>
</div>

<h3>Programmatically</h3>
<div class="code-block">
<div class="code-block-header">programmatic.ts <span class="lang-tag">ts</span></div>
Expand Down
82 changes: 82 additions & 0 deletions docs/record-replay/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -465,6 +465,88 @@ <h2>Fixture Auto-Generation</h2>
fixture is saved to disk with a warning but not registered in memory.
</p>

<h2 id="snapshot-style-recording">Snapshot-Style Recording</h2>
<p>
When the <code>X-Test-Id</code> header is present on a request, aimock uses
<strong>snapshot-style recording</strong> instead of the default timestamp-based
filenames. Fixtures are organized by test, producing stable file paths that work well with
version control and PR diffs.
</p>

<h3>Directory structure</h3>
<p>
The test ID is slugified into a directory name, and each provider gets its own file within
that directory:
</p>

<div class="code-block">
<div class="code-block-header">
Snapshot directory layout <span class="lang-tag">text</span>
</div>
<pre><code>fixtures/recorded/
agent-chat--handles-tool-call/
openai.json # All OpenAI fixtures for this test
anthropic.json # All Anthropic fixtures for this test
simple-test/
openai.json</code></pre>
</div>

<p>
The slugify rules: Common test file prefixes (<code>.spec.ts</code>,
<code>.test.tsx</code>, <code>.e2e.js</code>, etc.) are automatically stripped from the
test ID before slugifying, so <code>my-app.spec.ts › greeting</code> becomes
<code>greeting</code>. Then Playwright's <code>&nbsp;›&nbsp;</code> separator becomes
<code>--</code>, non-word characters become <code>-</code>, runs of 3+ dashes collapse to
<code>--</code>, and the result is lowercased. For example,
<code>"agent chat › handles tool call"</code> becomes
<code>agent-chat--handles-tool-call</code>.
</p>

<h3>Merge behavior on re-run</h3>
<p>
When you re-run a test, the new fixture is <strong>appended</strong> to the existing
<code>&lt;provider&gt;.json</code> file rather than overwriting it. This preserves
multi-turn conversations in a single file. If the existing file is corrupted (invalid
JSON), it is silently replaced.
</p>

<h3>Sending <code>X-Test-Id</code> from test frameworks</h3>

<div class="code-block">
<div class="code-block-header">Playwright <span class="lang-tag">ts</span></div>
<pre><code><span class="cm">// Playwright exposes testInfo.titlePath which joins suite + test titles</span>
<span class="kw">import</span> { test } <span class="kw">from</span> <span class="str">"@playwright/test"</span>;

test(<span class="str">"handles tool call"</span>, <span class="kw">async</span> ({ page }, testInfo) => {
<span class="cm">// titlePath = ["agent chat", "handles tool call"]</span>
<span class="kw">const</span> testId = testInfo.titlePath.join(<span class="str">" › "</span>);
<span class="cm">// Set on your OpenAI/Anthropic client config as a default header:</span>
<span class="cm">// headers: { "X-Test-Id": testId }</span>
});</code></pre>
</div>

<div class="code-block">
<div class="code-block-header">Vitest <span class="lang-tag">ts</span></div>
<pre><code><span class="kw">import</span> { describe, it } <span class="kw">from</span> <span class="str">"vitest"</span>;

describe(<span class="str">"agent chat"</span>, () => {
it(<span class="str">"handles tool call"</span>, <span class="kw">async</span> () => {
<span class="cm">// Pass X-Test-Id on each LLM request:</span>
<span class="kw">const</span> resp = <span class="kw">await</span> fetch(<span class="str">"http://localhost:4010/v1/chat/completions"</span>, {
headers: { <span class="str">"X-Test-Id"</span>: <span class="str">"agent chat › handles tool call"</span> },
<span class="cm">// ...body</span>
});
});
});</code></pre>
</div>

<h3>Fallback behavior</h3>
<p>
When no <code>X-Test-Id</code> header is present (or the value is
<code>__default__</code>), recording falls back to the standard timestamp-based filename:
<code>&lt;provider&gt;-&lt;timestamp&gt;-&lt;uuid&gt;.json</code>.
</p>

<h2>Fixture Lifecycle</h2>
<ul>
<li>
Expand Down
Loading
Loading