Skip to content

[AIRADSW-468] Fix Windows hang by draining the output pipe before waiting#4865

Open
ivarusic-amd wants to merge 10 commits into
developfrom
fix_win_hang
Open

[AIRADSW-468] Fix Windows hang by draining the output pipe before waiting#4865
ivarusic-amd wants to merge 10 commits into
developfrom
fix_win_hang

Conversation

@ivarusic-amd
Copy link
Copy Markdown
Contributor

@ivarusic-amd ivarusic-amd commented May 8, 2026

Motivation

Multiple models hanging on migraphx-driver compile step WINDOWS.

Technical Details

When migraphx-hiprtc-driver produced enough stdout/stderr to fill the Windows pipe buffer, the parent process would deadlock: child blocked writing to a full pipe, parent blocked in WaitForSingleObject waiting for the child to exit.
Fixed by draining the output pipe after closing stdin and before WaitForSingleObject

Linux should not be affected.

Changelog Category

Add a CHANGELOG.md entry for any option other than Not Applicable

    • Added: New functionality.
    • Changed: Changes to existing functionality.
    • Removed: Functionality or support that has been removed. (Compared to a previous release)
    • Optimized: Component performance that has been optimized or improved.
    • Resolved Issues: Known issues from a previous version that have been resolved.
    • Not Applicable: This PR is not to be included in the changelog.

@ivarusic-amd ivarusic-amd requested a review from causten as a code owner May 8, 2026 12:15
@ivarusic-amd ivarusic-amd added the Windows Related changes for Windows Environments label May 8, 2026
@apwojcik apwojcik added the UAI label May 8, 2026
@TedThemistokleous TedThemistokleous changed the title Fix Windows hang by draining the output pipe before waiting [AIRADSW-468] Fix Windows hang by draining the output pipe before waiting May 8, 2026
@causten
Copy link
Copy Markdown
Collaborator

causten commented May 8, 2026

How do you want to handle the warning? Empty While Loop. Without something the code will hold the CPU in a tight loop

ivarusic-amd and others added 2 commits May 11, 2026 13:15
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
@ivarusic-amd
Copy link
Copy Markdown
Contributor Author

@causten I change it, can you take a look. Thanks

@gh-app-migraphx-bot-pr-write
Copy link
Copy Markdown

gh-app-migraphx-bot-pr-write Bot commented May 11, 2026

Test Batch New Rate (e49c31) Old Rate (413902) Diff Status
torchvision-resnet50 64 2,092.75 2,180.39 -4.02%
torchvision-resnet50_fp16 64 6,529.97 2,618.95 149.34% 🔆
torchvision-densenet121 32 2,676.32 2,590.80 3.30%
torchvision-densenet121_fp16 32 4,540.30 4,556.87 -0.36%
torchvision-inceptionv3 32 1,712.17 1,797.28 -4.74%
torchvision-inceptionv3_fp16 32 2,814.20 2,826.73 -0.44%
cadene-inceptionv4 16 812.22 825.60 -1.62%
cadene-resnext64x4 16 550.17 784.52 -29.87% 🔴
slim-mobilenet 64 8,411.36 8,431.73 -0.24%
slim-nasnetalarge 64 225.73 229.17 -1.50%
slim-resnet50v2 64 1,952.86 2,839.08 -31.21% 🔴
bert-mrpc-onnx 8 508.39 1,170.04 -56.55% 🔴
bert-mrpc-tf 1 486.53 485.19 0.28%
pytorch-examples-wlang-gru 1 336.31 334.54 0.53%
pytorch-examples-wlang-lstm 1 466.80 460.96 1.27%
torchvision-resnet50_1 1 784.46 767.87 2.16%
cadene-dpn92_1 1 447.49 447.01 0.11%
cadene-resnext101_1 1 363.99 363.58 0.12%
onnx-taau-downsample 1 400.11 401.61 -0.37%
dlrm-criteoterabyte 1 28.78 32.69 -11.98% 🔴
dlrm-criteoterabyte_fp16 1 52.69 52.37 0.60%
agentmodel 1 10,435.74 9,867.77 5.76% 🔆
unet_fp16 2 54.39 21.15 157.17% 🔆
resnet50v1_fp16 1 941.28 922.03 2.09%
resnet50v1_int8 1 940.79 940.01 0.08%
bert_base_cased_fp16 64 1,103.54 1,103.36 0.02%
bert_large_uncased_fp16 32 347.74 347.72 0.01%
bert_large_fp16 1 124.75 204.50 -39.00% 🔴
distilgpt2_fp16 16 2,096.95 1,575.45 33.10% 🔆
yolov5s 1 573.38 466.21 22.99% 🔆
tinyllama 1 45.98 35.55 29.35% 🔆
vicuna-fastchat 1 44.04 43.74 0.68%
whisper-tiny-encoder 1 252.95 418.56 -39.57% 🔴
whisper-tiny-decoder 1 419.47 377.94 10.99% 🔆
llama2_7b 1 20.45 9.87 107.18% 🔆
qwen1.5-7b 1 23.68 23.61 0.28%
phi3-3.8b 1 26.83 26.82 0.04%
llama3-8b 1 21.83 21.78 0.21%
whisper-large-encoder 1 10.32 10.33 -0.08%
whisper-large-decoder 1 104.15 105.04 -0.85%
mistral-7b 1 23.82 13.41 77.63% 🔆
FLUX.1-schnell 1 746.65 793.08 -5.86% 🔴

Regressions detected 🔴

@gh-app-migraphx-bot-pr-write
Copy link
Copy Markdown

Test Status Result
bert-mrpc-onnx PASSED: MIGraphX meets tolerance
bert-mrpc-tf PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-gru PASSED: MIGraphX meets tolerance
pytorch-examples-wlang-lstm PASSED: MIGraphX meets tolerance
dlrm-criteoterabyte PASSED: MIGraphX meets tolerance
agentmodel PASSED: MIGraphX meets tolerance
unet PASSED: MIGraphX meets tolerance
resnet50v1 PASSED: MIGraphX meets tolerance
bert_base_cased_fp16 PASSED: MIGraphX meets tolerance
bert_large_uncased_fp16 🔴 FAILED: MIGraphX is not within tolerance - check verbose output
bert_large PASSED: MIGraphX meets tolerance
yolov5s PASSED: MIGraphX meets tolerance
tinyllama PASSED: MIGraphX meets tolerance
vicuna-fastchat PASSED: MIGraphX meets tolerance
whisper-tiny-encoder PASSED: MIGraphX meets tolerance
whisper-tiny-decoder PASSED: MIGraphX meets tolerance
distilgpt2_fp16 PASSED: MIGraphX meets tolerance
llama2_7b PASSED: MIGraphX meets tolerance
qwen1.5-7b PASSED: MIGraphX meets tolerance
phi3-3.8b PASSED: MIGraphX meets tolerance
llama3-8b PASSED: MIGraphX meets tolerance
whisper-large-encoder ERROR - check error output
traceback
Traceback (most recent call last):
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 377, in
main()
File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 224, in main
model = migraphx.parse_onnx(model_name, default_dim_value=batch)
RuntimeError: /data/src/include/migraphx/op/convolution.hpp:103: normalize_compute_shape: CONVOLUTION: mismatched channel numbers
whisper-large-decoder PASSED: MIGraphX meets tolerance
mistral-7b PASSED: MIGraphX meets tolerance
FLUX.1-schnell PASSED: MIGraphX meets tolerance

@CharlieL7 CharlieL7 removed their request for review May 12, 2026 20:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ok-to-test UAI Windows Related changes for Windows Environments

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants