Skip to content

Comments

[tests] Fix Windows test summary always reporting 'All tests passed'#24772

Open
rolfbjarne wants to merge 2 commits intomainfrom
dev/rolf/windows-not-all-rainbows
Open

[tests] Fix Windows test summary always reporting 'All tests passed'#24772
rolfbjarne wants to merge 2 commits intomainfrom
dev/rolf/windows-not-all-rainbows

Conversation

@rolfbjarne
Copy link
Member

@rolfbjarne rolfbjarne commented Feb 24, 2026

Fix a copy-paste bug in create-windows-html-report.cs where both the
success and failure branches wrote '# 🎉 All N tests passed 🎉'.
The failure branch now generates a proper summary with <details>/<summary>
tags listing failed tests, matching the format TestResults.psm1 expects.

Also fix TestResults.psm1 to emit <details> before <summary> (correct
HTML nesting order), and avoid including misleading success-format file
content in the failure section when the result file has no failure details.

Fix a copy-paste bug in create-windows-html-report.cs where both the
success and failure branches wrote '# 🎉 All N tests passed 🎉'.
The failure branch now generates a proper summary with <details>/<summary>
tags listing failed tests, matching the format TestResults.psm1 expects.

Also fix TestResults.psm1 to emit <details> before <summary> (correct
HTML nesting order), and avoid including misleading success-format file
content in the failure section when the result file has no failure details.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes the Windows test report summary generation so failures are correctly reported (instead of always saying “All tests passed”), and aligns summary formatting with what the PowerShell report aggregator expects.

Changes:

  • Update create-windows-html-report.cs to produce a failure summary using <details>/<summary> and list failed tests (or failed TRX files).
  • Fix TestResults.psm1 to emit valid HTML nesting (<details> before <summary>) and avoid echoing misleading “all tests passed” content in failure sections.
  • Update PowerShell tests to match the corrected output format.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File Description
tools/devops/automation/scripts/TestResults.psm1 Fixes <details>/<summary> emission order and improves failure-section content when the summary file lacks failure details.
tools/devops/automation/scripts/TestResults.Tests.ps1 Adjusts expected outputs to match the corrected HTML nesting and new failure messaging.
scripts/create-windows-html-report/create-windows-html-report.cs Generates a proper failure-mode markdown summary with <details>/<summary> and failed-test listings for Windows TRX parsing.
Comments suppressed due to low confidence (1)

scripts/create-windows-html-report/create-windows-html-report.cs:222

  • The failure summary writes trxName, test.Name, and the first line of test.Message directly into the markdown/HTML output. These values can contain characters like <, >, &, or newlines (\r\n) which can break the rendered report (and allow unintended HTML/markdown injection). Consider escaping/sanitizing these fields (and trimming \r) before appending them to summaryContents.
				foreach (var (trxName, test) in allFailedTests) {
					var msg = string.IsNullOrEmpty (test.Message) ? "" : $": {test.Message.Split ('\n') [0]}";
					summaryContents.AppendLine ($" * {trxName}/{test.Name}: {test.Outcome}{msg}");
				}

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

@vs-mobiletools-engineering-service2

This comment has been minimized.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@rolfbjarne rolfbjarne enabled auto-merge (squash) February 24, 2026 19:25
@vs-mobiletools-engineering-service2
Copy link
Collaborator

✅ [CI Build #9f585a6] Build passed (Build packages) ✅

Pipeline on Agent
Hash: 9f585a6ef854463964dbef77ca410e3c364266ad [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

✅ [PR Build #9f585a6] Build passed (Detect API changes) ✅

Pipeline on Agent
Hash: 9f585a6ef854463964dbef77ca410e3c364266ad [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

🔥 Unable to find the contents for the comment: D:\a\1\s\change-detection\results\gh-comment.md does not exist :fire

Pipeline on Agent
Hash: 9f585a6ef854463964dbef77ca410e3c364266ad [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

✅ [CI Build #9f585a6] Build passed (Build macOS tests) ✅

Pipeline on Agent
Hash: 9f585a6ef854463964dbef77ca410e3c364266ad [PR build]

@vs-mobiletools-engineering-service2
Copy link
Collaborator

🔥 [CI Build #9f585a6] Test results 🔥

Test results

❌ Tests failed on VSTS: test results

0 tests crashed, 2 tests failed, 154 tests passed.

Failures

❌ linker tests

2 tests failed, 42 tests passed.

Failed tests

  • link sdk/iOS - simulator/Debug: LaunchFailure
  • link all/iOS - simulator/Debug (don't bundle original resources): LaunchFailure

Html Report (VSDrops) Download

Successes

✅ cecil: All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (iOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (MacCatalyst): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (macOS): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (Multiple platforms): All 1 tests passed. Html Report (VSDrops) Download
✅ dotnettests (tvOS): All 1 tests passed. Html Report (VSDrops) Download
✅ framework: All 2 tests passed. Html Report (VSDrops) Download
✅ fsharp: All 4 tests passed. Html Report (VSDrops) Download
✅ generator: All 5 tests passed. Html Report (VSDrops) Download
✅ interdependent-binding-projects: All 4 tests passed. Html Report (VSDrops) Download
✅ introspection: All 6 tests passed. Html Report (VSDrops) Download
✅ monotouch (iOS): All 11 tests passed. Html Report (VSDrops) Download
✅ monotouch (MacCatalyst): All 15 tests passed. Html Report (VSDrops) Download
✅ monotouch (macOS): All 12 tests passed. Html Report (VSDrops) Download
✅ monotouch (tvOS): All 11 tests passed. Html Report (VSDrops) Download
✅ msbuild: All 2 tests passed. Html Report (VSDrops) Download
✅ sharpie: All 1 tests passed. Html Report (VSDrops) Download
✅ windows: All 3 tests passed. Html Report (VSDrops) Download
✅ xcframework: All 4 tests passed. Html Report (VSDrops) Download
✅ xtro: All 1 tests passed. Html Report (VSDrops) Download

macOS tests

✅ Tests on macOS Monterey (12): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Ventura (13): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sonoma (14): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Sequoia (15): All 5 tests passed. Html Report (VSDrops) Download
✅ Tests on macOS Tahoe (26): All 5 tests passed. Html Report (VSDrops) Download

Pipeline on Agent
Hash: 9f585a6ef854463964dbef77ca410e3c364266ad [PR build]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants