Skip to content

Fix: Define tempFolder in Windows CI to prevent false-positive test results#1074

Open
x15sr71 wants to merge 1 commit intoCCExtractor:masterfrom
x15sr71:master
Open

Fix: Define tempFolder in Windows CI to prevent false-positive test results#1074
x15sr71 wants to merge 1 commit intoCCExtractor:masterfrom
x15sr71:master

Conversation

@x15sr71
Copy link

@x15sr71 x15sr71 commented Mar 22, 2026

In raising this pull request, I confirm the following (please check boxes):

  • I have read and understood the contributors guide.
  • I have checked that another pull request for this purpose does not exist.
  • I have considered, and confirmed that this submission will be valuable to others.
  • I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
  • I give this submission freely, and claim no ownership to its content.

My familiarity with the project is as follows (check one):

  • I have never used the project.
  • I have used the project briefly.
  • I have used the project extensively, but have not contributed previously.
  • I am an active contributor to the project.

Problem

variables.bat never defines tempFolder, so %tempFolder% expands to "" when runCI.bat invokes ccextractortester with --tempfolder "%tempFolder%".

This triggers a silent false-positive chain in ccx_testsuite:

  • Path.Combine("", expectedFile) → bare filename with no folder
  • FileInfo(bare).Existsfalse"" stored as produced path (Processor.cs -> ccx_testsuite)
  • File.Open("") throws ArgumentException in ServerComparer.cs
  • catch(Exception) only logs it, skips comparison entirely (Tester.cs)
  • SendExitCodeAndRuntime fires with exitCode=0 → server receives exit code but no comparison result, platform interprets absence of failure as pass

Result: affected tests on Windows pass without any comparison being executed.

Confirmed across live test runs on the same commit (03ad9e8):

  • Linux run 8976: SPUPNG and other tests correctly show Fail
  • Windows run 8977: same tests show Pass, with multiple [ERROR] Empty path name is not legal. entries followed by exit code: 0 in the log

This explains a persistent pattern across multiple runs — all Linux/Windows result mismatches are in the same direction:
Linux correctly fails, Windows silently passes. Across two sampled runs (9614f58: 9 mismatches, 03ad9e8: 7 mismatches), zero Windows-only failures exist despite both platforms using the same golden files and the same CCExtractor commit.

Linux works correctly because variables already defines: tempFolder="/repository/TempFiles"

Fix

Add the missing variable to variables.bat:

set tempFolder=.\TempFiles

Testing

  • Verified via [System.IO.Path]::Combine behavior: Path.Combine("", expectedFile) resolves to a bare filename (unfindable), while Path.Combine(".\TempFiles", expectedFile) resolves to a valid path (e.g. .\TempFiles\file.xml)
  • Confirmed from Processor.cs (firstOutputFileFI.Directory.Create()) that .\TempFiles is created automatically before CCExtractor runs
  • Expected behavior: Windows test runs should no longer show Empty path name is not legal, and comparisons should be executed as intended, and genuine failures should be correctly reported as Fail instead of Pass.

Impact

  • Ensures output files are written to a predictable location
  • Allows the tester to correctly locate and compare outputs
  • Eliminates silent false-positive test passes on Windows
  • Aligns Windows CI behavior with Linux

Relevant Zulip thread

@sonarqubecloud
Copy link

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant