Skip to content

test(dashboard): wait for page load before asserting in annotate tests#40739

Closed
SebTardif wants to merge 1 commit intomicrosoft:mainfrom
SebTardif:fix/annotate-test-wait-for-load
Closed

test(dashboard): wait for page load before asserting in annotate tests#40739
SebTardif wants to merge 1 commit intomicrosoft:mainfrom
SebTardif:fix/annotate-test-wait-for-load

Conversation

@SebTardif
Copy link
Copy Markdown
Contributor

Summary

Add waitForLoadState('load') after obtaining the dashboard page in all 13 annotate test sites. The tests grab the page via browser.contexts()[0].pages()[0] and immediately assert or click without waiting for the page to finish loading.

Problem

The annotate tests are the dominant source of MCP CI failures on Windows. Every failing test shows the same error in CI logs:

Locator: getByRole('main', { name: 'Dashboard: annotate' })
Expected: visible
Received: undefined
Timeout: 5000ms
Call log:
  - waiting for "http://localhost:XXXXX/" navigation to finish...

The page hasn't finished navigating when the assertion starts. On Linux/macOS, the dashboard loads within the 5000ms default expect timeout. On Windows CI, it doesn't.

CI failure statistics (last 14 MCP workflow runs)

Data from runs 25543907347 through 25579035284 (May 8, 2026):

Metric Value
MCP workflow runs analyzed 14 failed + 1 passed + 5 cancelled/pending
Total individual test failures 90
Annotate test failures 47 (52% of all failures)
Runs with at least one annotate failure 12 / 14 (86%)
Runs with navigation to finish in logs 9 / 14 (64%)
Runs that would be fully green without annotate failures 1 / 14

The annotate failures are spread across all Windows browser projects (chrome, chromium, msedge, webkit, firefox) and hit nearly every annotate test:

Test Failures across 14 runs
should abort MCP annotation when last screenshot is removed 6
should capture multiple screenshots in one annotation 5
should cancel browser_annotate when the MCP request is aborted 5
should enter annotate mode on fresh dashboard.tsx mount 5
user-initiated annotate downloads zip with feedback.md 5
should abort annotation when last screenshot is removed 4
should cancel browser_annotate when the MCP client disconnects 3
should annotate via direct browser_annotate MCP call 3
should annotate when context has no fixed viewport 3

Root cause

Every annotate test follows this pattern:

const browser = await connectToDashboard(bindTitle);
const dashboard = browser.contexts()[0].pages()[0];
// Immediately assert or click without waiting for page load:
await expect(dashboard.getByRole('main', { name: 'Dashboard: annotate' })).toBeVisible();

connectToDashboard connects to the Playwright server but does not wait for any page to load. The test grabs the first page and immediately starts asserting. Playwright's expect().toBeVisible() retries within its 5000ms timeout, but if the page navigation itself hasn't completed (the dashboard is a React SPA that needs to load HTML, JS bundles, and render), the locator search fails because the DOM is empty.

The McpError: Connection closed errors reported alongside these failures are a consequence, not the cause: when the assertion times out and the test fails, the framework kills the MCP server during cleanup, which closes the connection on the pending browser_annotate call.

Fix

Add await dashboard.waitForLoadState('load') after every browser.contexts()[0].pages()[0] call (13 sites). This ensures the page HTML and scripts have loaded before any assertions or interactions begin.

Related: #40729 (bumps timeout for one specific assertion), #40348 (fixes a different annotate race), #40296 (attempted IPC fix, closed).

On Windows CI, the dashboard page navigation can take longer than the
default 5000ms expect timeout. Every annotate test grabs the dashboard
page via browser.contexts()[0].pages()[0] and immediately asserts or
clicks without waiting for the page to finish loading.

Add waitForLoadState('load') after obtaining the dashboard page in all
13 test sites. This ensures the page HTML and scripts have loaded before
any assertions run.
@pavelfeldman
Copy link
Copy Markdown
Member

That should not be necessary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants