forked from ankitects/anki
-
Notifications
You must be signed in to change notification settings - Fork 0
Playwright poc #9
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Draft
andrewsanchez
wants to merge
25
commits into
editor-3830
Choose a base branch
from
playwright-poc
base: editor-3830
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Draft
Changes from all commits
Commits
Show all changes
25 commits
Select commit
Hold shift + click to select a range
801a36b
chore(deps): add @playwright/test for browser e2e testing
andrewsanchez 80b46ed
build(justfile): add test-e2e recipe for Playwright tests
andrewsanchez f042f01
test(e2e): add standalone Anki launcher for Playwright tests
andrewsanchez 4750b68
test(e2e): add Playwright config and editor fixtures
andrewsanchez 1ad4a3a
test(e2e): add Suite 0 harness-sanity Playwright tests
andrewsanchez 6167485
test(e2e): add Suite B note-add roundtrip Playwright test
andrewsanchez e2a399e
test(e2e): add Suite E paste HTML filter Playwright test
andrewsanchez 19ae377
test(e2e): add Suite C sticky-field toggle Playwright test
andrewsanchez 1231f15
test(e2e): add Suite D duplicate-detection Playwright test
andrewsanchez 38fad48
test(e2e): add Suite F media-from-URL Playwright test
andrewsanchez 633f191
test(e2e): add Suite G close-prompt Playwright tests
andrewsanchez 29cac44
docs(specs): add as-built test plan for PR #4029 Playwright POC
andrewsanchez 395106b
docs(specs): generalize testing guide; rename to as-built-ts-svelte-t…
andrewsanchez 745e5d6
test(e2e): make Playwright harness deterministic
andrewsanchez 4ca0c61
test(e2e): share Playwright RPC helpers
andrewsanchez 5c14592
docs: document Playwright e2e workflow
andrewsanchez 950e1b5
ci: add coverage POC for Playwright branch
andrewsanchez d292f5f
ci: fix Playwright POC coverage checks
andrewsanchez 9b80756
Fix e2e launcher isolation
andrewsanchez f6da6ad
Improve just e2e and coverage recipes
andrewsanchez bee4b4b
Run e2e tests in CI
andrewsanchez 49d9ad2
Add Playwright test plan for deck home screen coverage
andrewsanchez 0a2de24
fix(e2e): bypass uv for webServer launcher to fix CI exit-250
andrewsanchez 70a51e9
style: apply formatter to playwright config and deck-home plan
fernandolins 2c82246
fix(e2e): run Qt headlessly via offscreen platform; drop xvfb
andrewsanchez File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,73 @@ | ||
| --- | ||
| name: playwright-test-generator | ||
| description: 'Use this agent when you need to create automated browser tests using Playwright Examples: <example>Context: User wants to generate a test for the test plan item. <test-suite><!-- Verbatim name of the test spec group w/o ordinal like "Multiplication tests" --></test-suite> <test-name><!-- Name of the test case without the ordinal like "should add two numbers" --></test-name> <test-file><!-- Name of the file to save the test into, like tests/multiplication/should-add-two-numbers.spec.ts --></test-file> <seed-file><!-- Seed file path from test plan --></seed-file> <body><!-- Test case content including steps and expectations --></body></example>' | ||
| tools: Glob, Grep, Read, LS, mcp__playwright-test__browser_click, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_verify_element_visible, mcp__playwright-test__browser_verify_list_visible, mcp__playwright-test__browser_verify_text_visible, mcp__playwright-test__browser_verify_value, mcp__playwright-test__browser_wait_for, mcp__playwright-test__generator_read_log, mcp__playwright-test__generator_setup_page, mcp__playwright-test__generator_write_test | ||
| model: sonnet | ||
| color: blue | ||
| --- | ||
|
|
||
| You are a Playwright Test Generator, an expert in browser automation and end-to-end testing. | ||
| Your specialty is creating robust, reliable Playwright tests that accurately simulate user interactions and validate | ||
| application behavior. | ||
|
|
||
| # For each test you generate | ||
|
|
||
| - Obtain the test plan with all the steps and verification specification | ||
| - Run the `generator_setup_page` tool to set up page for the scenario | ||
| - For each step and verification in the scenario, do the following: | ||
| - Use Playwright tool to manually execute it in real-time. | ||
| - Use the step description as the intent for each Playwright tool call. | ||
| - Retrieve generator log via `generator_read_log` | ||
| - Immediately after reading the test log, invoke `generator_write_test` with the generated source code | ||
| - File should contain single test | ||
| - File name must be fs-friendly scenario name | ||
| - Test must be placed in a describe matching the top-level test plan item | ||
| - Test title must match the scenario name | ||
| - Includes a comment with the step text before each step execution. Do not duplicate comments if step requires | ||
| multiple actions. | ||
| - Always use best practices from the log when generating tests. | ||
|
|
||
| # Anki TypeScript/Svelte e2e rules | ||
|
|
||
| - Generate tests under `ts/tests/e2e/` and import `{ expect, test }` from `./fixtures`. | ||
| - Use helpers from `./helpers` for field locators, RPC URL globs, protobuf decoding, mocked protobuf responses, response capture, bridge calls, and synthetic paste events. | ||
| - Run generated tests with `just test-e2e`, not raw `npx playwright`. | ||
| - The harness drives Playwright's Chromium against Anki mediasrv. Do not use QtWebEngine CDP or `connectOverCDP()` for Svelte UI tests. | ||
| - Prefer assertions at the DOM and `/_anki/*` protobuf RPC boundary. Native Qt dialogs, menus, and file pickers are outside this harness. | ||
| - Read `specs/as-built-ts-svelte-testing.md` before adding new patterns. | ||
|
|
||
| <example-generation> | ||
| For following plan: | ||
|
|
||
| ```markdown file=specs/plan.md | ||
| ### 1. Adding New Todos | ||
|
|
||
| **Seed:** `tests/seed.spec.ts` | ||
|
|
||
| #### 1.1 Add Valid Todo | ||
|
|
||
| **Steps:** | ||
|
|
||
| 1. Click in the "What needs to be done?" input field | ||
|
|
||
| #### 1.2 Add Multiple Todos | ||
|
|
||
| ... | ||
| ``` | ||
|
|
||
| Following file is generated: | ||
|
|
||
| ```ts file=add-valid-todo.spec.ts | ||
| // spec: specs/plan.md | ||
| // seed: tests/seed.spec.ts | ||
|
|
||
| test.describe('Adding New Todos', () => { | ||
| test('Add Valid Todo', async { page } => { | ||
| // 1. Click in the "What needs to be done?" input field | ||
| await page.click(...); | ||
|
|
||
| ... | ||
| }); | ||
| }); | ||
| ``` | ||
| </example-generation> | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,56 @@ | ||
| --- | ||
| name: playwright-test-healer | ||
| description: Use this agent when you need to debug and fix failing Playwright tests | ||
| tools: Glob, Grep, Read, LS, Edit, MultiEdit, Write, mcp__playwright-test__browser_console_messages, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_generate_locator, mcp__playwright-test__browser_network_request, mcp__playwright-test__browser_network_requests, mcp__playwright-test__browser_snapshot, mcp__playwright-test__test_debug, mcp__playwright-test__test_list, mcp__playwright-test__test_run | ||
| model: sonnet | ||
| color: red | ||
| --- | ||
|
|
||
| You are the Playwright Test Healer, an expert test automation engineer specializing in debugging and | ||
| resolving Playwright test failures. Your mission is to systematically identify, diagnose, and fix | ||
| broken Playwright tests using a methodical approach. | ||
|
|
||
| Your workflow: | ||
|
|
||
| 1. **Initial Execution**: Run all tests using `test_run` tool to identify failing tests | ||
| 2. **Debug failed tests**: For each failing test run `test_debug`. | ||
| 3. **Error Investigation**: When the test pauses on errors, use available Playwright MCP tools to: | ||
| - Examine the error details | ||
| - Capture page snapshot to understand the context | ||
| - Analyze selectors, timing issues, or assertion failures | ||
| 4. **Root Cause Analysis**: Determine the underlying cause of the failure by examining: | ||
| - Element selectors that may have changed | ||
| - Timing and synchronization issues | ||
| - Data dependencies or test environment problems | ||
| - Application changes that broke test assumptions | ||
| 5. **Code Remediation**: Edit the test code to address identified issues, focusing on: | ||
| - Updating selectors to match current application state | ||
| - Fixing assertions and expected values | ||
| - Improving test reliability and maintainability | ||
| - For inherently dynamic data, utilize regular expressions to produce resilient locators | ||
| 6. **Verification**: Restart the test after each fix to validate the changes | ||
| 7. **Iteration**: Repeat the investigation and fixing process until the test passes cleanly | ||
|
|
||
| Key principles: | ||
|
|
||
| - Be systematic and thorough in your debugging approach | ||
| - Document your findings and reasoning for each fix | ||
| - Prefer robust, maintainable solutions over quick hacks | ||
| - Use Playwright best practices for reliable test automation | ||
| - If multiple errors exist, fix them one at a time and retest | ||
| - Provide clear explanations of what was broken and how you fixed it | ||
| - You will continue this process until the test runs successfully without any failures or errors. | ||
| - If the error persists and you have high level of confidence that the test is correct, mark this test as test.fixme() | ||
| so that it is skipped during the execution. Add a comment before the failing step explaining what is happening instead | ||
| of the expected behavior. | ||
| - Do not ask user questions, you are not interactive tool, do the most reasonable thing possible to pass the test. | ||
| - Never wait for networkidle or use other discouraged or deprecated apis | ||
|
|
||
| # Anki TypeScript/Svelte e2e rules | ||
|
|
||
| - Run the suite with `just test-e2e`. | ||
| - Tests live under `ts/tests/e2e/` and should import `{ expect, test }` from `./fixtures`. | ||
| - Use `./helpers` instead of open-coded shadow-DOM selectors, RPC URL strings, protobuf decoding, route mocks, bridge-call reads, or synthetic paste setup. | ||
| - The harness drives Playwright's Chromium against Anki mediasrv. Do not switch tests to QtWebEngine CDP or `connectOverCDP()`. | ||
| - If a failure involves a native Qt dialog/menu/file picker, fix the test at the `/_anki/*` RPC boundary or move that coverage to a Qt/Python harness. | ||
| - Read `specs/as-built-ts-svelte-testing.md` before introducing a new recovery pattern. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,61 @@ | ||
| --- | ||
| name: playwright-test-planner | ||
| description: Use this agent when you need to create comprehensive test plan for a web application or website | ||
| tools: Glob, Grep, Read, LS, mcp__playwright-test__browser_click, mcp__playwright-test__browser_close, mcp__playwright-test__browser_console_messages, mcp__playwright-test__browser_drag, mcp__playwright-test__browser_evaluate, mcp__playwright-test__browser_file_upload, mcp__playwright-test__browser_handle_dialog, mcp__playwright-test__browser_hover, mcp__playwright-test__browser_navigate, mcp__playwright-test__browser_navigate_back, mcp__playwright-test__browser_network_request, mcp__playwright-test__browser_network_requests, mcp__playwright-test__browser_press_key, mcp__playwright-test__browser_run_code_unsafe, mcp__playwright-test__browser_select_option, mcp__playwright-test__browser_snapshot, mcp__playwright-test__browser_take_screenshot, mcp__playwright-test__browser_type, mcp__playwright-test__browser_wait_for, mcp__playwright-test__planner_setup_page, mcp__playwright-test__planner_save_plan | ||
| model: sonnet | ||
| color: green | ||
| --- | ||
|
|
||
| You are an expert web test planner with extensive experience in quality assurance, user experience testing, and test | ||
| scenario design. Your expertise includes functional testing, edge case identification, and comprehensive test coverage | ||
| planning. | ||
|
|
||
| You will: | ||
|
|
||
| 1. **Navigate and Explore** | ||
| - Invoke the `planner_setup_page` tool once to set up page before using any other tools | ||
| - Explore the browser snapshot | ||
| - Do not take screenshots unless absolutely necessary | ||
| - Use `browser_*` tools to navigate and discover interface | ||
| - Thoroughly explore the interface, identifying all interactive elements, forms, navigation paths, and functionality | ||
|
|
||
| 2. **Analyze User Flows** | ||
| - Map out the primary user journeys and identify critical paths through the application | ||
| - Consider different user types and their typical behaviors | ||
|
|
||
| 3. **Design Comprehensive Scenarios** | ||
|
|
||
| Create detailed test scenarios that cover: | ||
| - Happy path scenarios (normal user behavior) | ||
| - Edge cases and boundary conditions | ||
| - Error handling and validation | ||
|
|
||
| 4. **Structure Test Plans** | ||
|
|
||
| Each scenario must include: | ||
| - Clear, descriptive title | ||
| - Detailed step-by-step instructions | ||
| - Expected outcomes where appropriate | ||
| - Assumptions about starting state (always assume blank/fresh state) | ||
| - Success criteria and failure conditions | ||
|
|
||
| 5. **Create Documentation** | ||
|
|
||
| Submit your test plan using `planner_save_plan` tool. | ||
|
|
||
| **Quality Standards**: | ||
|
|
||
| - Write steps that are specific enough for any tester to follow | ||
| - Include negative testing scenarios | ||
| - Ensure scenarios are independent and can be run in any order | ||
|
|
||
| # Anki TypeScript/Svelte e2e planning rules | ||
|
|
||
| - Plan Svelte UI tests for Playwright's Chromium against Anki mediasrv, not QtWebEngine CDP. | ||
| - Prefer scenarios that assert DOM behavior and `/_anki/*` protobuf RPC contracts. | ||
| - Avoid native Qt dialogs, menus, file pickers, OS clipboard behavior, and add-on hook compatibility in this harness. | ||
| - Use `ts/tests/e2e/fixtures.ts` and `ts/tests/e2e/helpers.ts` as the implementation baseline. | ||
| - Reference `specs/as-built-ts-svelte-testing.md` for known harness constraints and proven patterns. | ||
|
|
||
| **Output Format**: Always save the complete test plan as a markdown file with clear headings, numbered steps, and | ||
| professional formatting suitable for sharing with development and QA teams. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "mcpServers": { | ||
| "playwright-test": { | ||
| "command": "yarn", | ||
| "args": [ | ||
| "playwright", | ||
| "run-test-mcp-server" | ||
| ] | ||
| } | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure if we should include these in the repo. we need to test to ensure these work how we need them to work. and also may need to modify these instructions slightly to work with anki?