Fix: Define tempFolder in Windows CI to prevent false-positive test results#1074
Open
x15sr71 wants to merge 1 commit intoCCExtractor:masterfrom
Open
Fix: Define tempFolder in Windows CI to prevent false-positive test results#1074x15sr71 wants to merge 1 commit intoCCExtractor:masterfrom
x15sr71 wants to merge 1 commit intoCCExtractor:masterfrom
Conversation
…ositive test passes
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



In raising this pull request, I confirm the following (please check boxes):
My familiarity with the project is as follows (check one):
Problem
variables.batnever definestempFolder, so%tempFolder%expands to""whenrunCI.batinvokesccextractortesterwith--tempfolder "%tempFolder%".This triggers a silent false-positive chain in
ccx_testsuite:Path.Combine("", expectedFile)→ bare filename with no folderFileInfo(bare).Exists→false→""stored as produced path (Processor.cs->ccx_testsuite)File.Open("")throwsArgumentExceptioninServerComparer.cscatch(Exception)only logs it, skips comparison entirely (Tester.cs)SendExitCodeAndRuntimefires withexitCode=0→ server receives exit code but no comparison result, platform interprets absence of failure as passResult: affected tests on Windows pass without any comparison being executed.
Confirmed across live test runs on the same commit (03ad9e8):
[ERROR] Empty path name is not legal.entries followed byexit code: 0in the logThis explains a persistent pattern across multiple runs — all Linux/Windows result mismatches are in the same direction:
Linux correctly fails, Windows silently passes. Across two sampled runs (9614f58: 9 mismatches, 03ad9e8: 7 mismatches), zero Windows-only failures exist despite both platforms using the same golden files and the same CCExtractor commit.
Linux works correctly because
variablesalready defines:tempFolder="/repository/TempFiles"Fix
Add the missing variable to
variables.bat:Testing
Path.Combine("", expectedFile)resolves to a bare filename (unfindable), whilePath.Combine(".\TempFiles", expectedFile)resolves to a valid path (e.g..\TempFiles\file.xml)Processor.cs(firstOutputFileFI.Directory.Create()) that.\TempFilesis created automatically before CCExtractor runsEmpty path name is not legal, and comparisons should be executed as intended, and genuine failures should be correctly reported as Fail instead of Pass.Impact
Relevant Zulip thread