Skip to content

fix(ci): create tracesDir before stacks write + add MCP test retries#40735

Closed
SebTardif wants to merge 2 commits intomicrosoft:mainfrom
SebTardif:fix/mcp-test-ci-retries
Closed

fix(ci): create tracesDir before stacks write + add MCP test retries#40735
SebTardif wants to merge 2 commits intomicrosoft:mainfrom
SebTardif:fix/mcp-test-ci-retries

Conversation

@SebTardif
Copy link
Copy Markdown
Contributor

@SebTardif SebTardif commented May 8, 2026

Summary

  • Fix ENOENT when writing .stacks file during live tracing (the directory may not exist yet)
  • Add retries: process.env.CI ? 3 : 0 to the MCP test config, matching every other test suite

Bug: ENOENT on .stacks file (tracing.spec.ts)

tracingStarted() in localUtils.ts computes the stacks file path inside tracesDir, but when tracesDir is provided (not a temp dir), the directory is never created. The zip() function does mkdir(path.dirname(...), { recursive: true }) before writing the trace zip, but addStackToTracingNoReply() assumes the directory exists and calls writeFile() directly.

On Windows CI, this causes an intermittent ENOENT because the directory creation timing between the browser launch (which creates tracesDir for its own trace files) and the first client-side stacks write is less predictable:

Error: ENOENT: no such file or directory, open
  'D:\...\test-results\tracing-...-firefox\.playwright-mcp\traces\trace-1778265446373.stacks'
    at async open (node:internal/fs/promises:639:25)
    at async Object.writeFile (node:internal/fs/promises:1219:14)

Fix: call mkdir({ recursive: true }) in tracingStarted() when tracesDir is provided, so the directory is guaranteed to exist before any stacks are written. This matches the existing pattern in zip().

Example CI failure: PR #40722, windows-latest - firefox at tracing.spec.ts:54.

Retries: match other test configs

The MCP test suite is the only test config in the project without CI retries:

Config Retries
tests/library/playwright.config.ts process.env.CI ? 3 : 0
tests/installation/playwright.config.ts process.env.CI ? 3 : 0
tests/electron/playwright.config.ts process.env.CI ? 3 : 0
tests/extension/playwright.config.ts process.env.CI ? 2 : 0
tests/android/playwright.config.ts process.env.CI ? 1 : 0
tests/mcp/playwright.config.ts none

The ENOENT bug is a concrete fix, but the MCP tests also have non-deterministic failures that only retries can address:

  • McpError: Connection closed during teardown races (e.g., config.ini.spec.ts, video.spec.ts on webkit Windows)
  • Flaky toHaveCount assertions on timing-sensitive UI elements (e.g., annotate.spec.ts)
  • HTTP transport lifecycle races (e.g., http.spec.ts on webkit Windows)

These are the same class of non-deterministic failures that retries already handle in every other test suite.

SebTardif added 2 commits May 8, 2026 13:32
When tracesDir is provided (not a temp dir), the directory may not
exist yet when the first live stack is written. The zip() function
already does mkdir before writing, but addStackToTracingNoReply
assumes the directory exists and fails with ENOENT on Windows CI
where directory creation timing is less predictable.
@SebTardif SebTardif changed the title fix(ci): add retries for MCP tests to match other test configs fix(ci): create tracesDir before stacks write + add MCP test retries May 8, 2026
@SebTardif
Copy link
Copy Markdown
Contributor Author

Closing: the .stacks mkdir fix duplicates #40730, and the retries alone don't address the root cause (Connection closed in annotate tests on Windows). The team is already actively working on these issues.

@SebTardif SebTardif closed this May 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant