[AIMIGRAPHX-580] [AIMIGRAPHX-623] jit implementation for nonzero and scan lib#4653
Draft
[AIMIGRAPHX-580] [AIMIGRAPHX-623] jit implementation for nonzero and scan lib#4653
Conversation
bdevorem
changed the title
bdevorem
force-pushed
the
bdevorem/AIMIGRAPHX-580-jit-implementation-for-nonzero
branch
from
c3d0e48 to
1863f18
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR migrates the GPU nonzero operator from the legacy device implementation to a HIP JIT-compiled kernel, and introduces a kernels-level scan utility to support that implementation.
Changes:
- Removed the legacy GPU
nonzeroop wiring and device implementation, switching lowering to use the JIT compiler path. - Added new JIT compiler for
nonzeroplus corresponding kernels headers (nonzero.hpp,scan.hpp). - Updated build registration and changelog to reflect the conversion.
Reviewed changes
Copilot reviewed 10 out of 10 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| src/targets/gpu/nonzero.cpp | Removes the legacy hip_nonzero op implementation entrypoint. |
| src/targets/gpu/lowering.cpp | Stops lowering nonzero via add_extend_op, letting the compiler/JIT path handle it. |
| src/targets/gpu/kernels/include/migraphx/kernels/scan.hpp | Adds kernels-level block_scan/reverse_scan utilities used by JIT kernels. |
| src/targets/gpu/kernels/include/migraphx/kernels/nonzero.hpp | Adds kernels-level nonzero implementation built on block_scan. |
| src/targets/gpu/jit/nonzero.cpp | Adds the HIP JIT compiler that emits/compiles the nonzero kernel. |
| src/targets/gpu/include/migraphx/gpu/nonzero.hpp | Removes the legacy hip_nonzero op declaration. |
| src/targets/gpu/include/migraphx/gpu/device/nonzero.hpp | Removes the legacy device API declaration for nonzero. |
| src/targets/gpu/device/nonzero.cpp | Removes the legacy device kernel launcher for nonzero. |
| src/targets/gpu/CMakeLists.txt | Removes legacy sources/op registration and relies on JIT source globbing. |
| CHANGELOG.md | Notes the nonzero migration from device to JIT. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
bdevorem
force-pushed
the
bdevorem/AIMIGRAPHX-580-jit-implementation-for-nonzero
branch
from
5664be8 to
4defa9d
Compare
anisha-amd
approved these changes
Mar 9, 2026
Collaborator
This build is not recommended to merge 🔴 |
Collaborator
❌bert_base_cased_fp16: ERROR - check error output�[1;31m2026-03-09 19:34:44.740478456 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075389, index: 63, mask: {64, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m�[1;31m2026-03-09 19:34:44.749903598 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075416, index: 90, mask: {91, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749921873 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075391, index: 65, mask: {66, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753897375 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075395, index: 69, mask: {70, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753908677 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075404, index: 78, mask: {79, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753915840 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075411, index: 85, mask: {86, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753931800 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075407, index: 81, mask: {82, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749915871 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075417, index: 91, mask: {92, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749915891 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075406, index: 80, mask: {81, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749922965 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075401, index: 75, mask: {76, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749924508 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075398, index: 72, mask: {73, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749928846 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075397, index: 71, mask: {72, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749923215 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075424, index: 98, mask: {99, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.757897083 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075428, index: 102, mask: {103, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749906484 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075399, index: 73, mask: {74, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749941009 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075412, index: 86, mask: {87, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749942712 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075419, index: 93, mask: {94, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749948984 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075427, index: 101, mask: {102, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753898357 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075396, index: 70, mask: {71, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753904940 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075410, index: 84, mask: {85, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753906082 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075429, index: 103, mask: {104, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753913416 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075430, index: 104, mask: {105, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753913225 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075408, index: 82, mask: {83, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749901554 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075390, index: 64, mask: {65, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753918134 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075422, index: 96, mask: {97, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753915870 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075423, index: 97, mask: {98, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753920619 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075413, index: 87, mask: {88, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753922503 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075402, index: 76, mask: {77, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753923805 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075414, index: 88, mask: {89, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753923775 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075418, index: 92, mask: {93, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749901574 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075392, index: 66, mask: {67, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753928123 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075403, index: 77, mask: {78, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753928173 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075421, index: 95, mask: {96, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753928153 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075409, index: 83, mask: {84, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753931820 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075415, index: 89, mask: {90, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749939576 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075425, index: 99, mask: {100, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753938553 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075405, index: 79, mask: {80, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749908047 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075394, index: 68, mask: {69, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.766730660 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075448, index: 122, mask: {123, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.766855876 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075434, index: 108, mask: {109, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.766918705 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075447, index: 121, mask: {122, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.767109314 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075440, index: 114, mask: {115, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.767251682 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075439, index: 113, mask: {114, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753965965 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075420, index: 94, mask: {95, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.757904056 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075431, index: 105, mask: {106, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.749901534 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075393, index: 67, mask: {68, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.767257393 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075446, index: 120, mask: {121, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.767658068 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075438, index: 112, mask: {113, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.767674198 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075445, index: 119, mask: {120, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.767693034 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075452, index: 126, mask: {127, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.767721277 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075433, index: 107, mask: {108, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753937671 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075426, index: 100, mask: {101, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768325275 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075437, index: 111, mask: {112, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768335154 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075435, index: 109, mask: {110, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768368186 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075444, index: 118, mask: {119, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768368657 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075442, index: 116, mask: {117, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768354170 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075451, index: 125, mask: {126, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768421176 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075449, index: 123, mask: {124, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768539108 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075432, index: 106, mask: {107, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768564125 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075436, index: 110, mask: {111, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768596005 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075443, index: 117, mask: {118, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768736240 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075450, index: 124, mask: {125, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.768977825 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075441, index: 115, mask: {116, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:44.753949433 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2075400, index: 74, mask: {75, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:34:46.265910971 [E:onnxruntime:, inference_session.cc:2544 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/graph/graph_utils.cc:29 int onnxruntime::graph_utils::GetIndexFromName(const onnxruntime::Node&, const std::string&, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_onnx::Pow_1623for node: Mul_53/SimplifiedLayerNormFusion/ �[m Traceback (most recent call last): File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 359, in main() File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 278, in main sess = ort.InferenceSession(model_name, File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 485, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 584, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/graph/graph_utils.cc:29 int onnxruntime::graph_utils::GetIndexFromName(const onnxruntime::Node&, const std::string&, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_onnx::Pow_1623for node: Mul_53/SimplifiedLayerNormFusion/ ❌bert_large_uncased_fp16: ERROR - check error output�[1;31m2026-03-09 19:35:31.885505433 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098241, index: 63, mask: {64, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m�[1;31m2026-03-09 19:35:31.893916334 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098243, index: 65, mask: {66, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897896275 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098247, index: 69, mask: {70, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897915451 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098245, index: 67, mask: {68, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897903148 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098242, index: 64, mask: {65, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897904159 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098252, index: 74, mask: {75, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897905833 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098246, index: 68, mask: {69, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897918777 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098251, index: 73, mask: {74, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.901893739 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098248, index: 70, mask: {71, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.905889909 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098250, index: 72, mask: {73, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897923626 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098249, index: 71, mask: {72, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897929607 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098253, index: 75, mask: {76, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.897902957 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098244, index: 66, mask: {67, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.911864749 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098259, index: 81, mask: {82, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.911879347 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098266, index: 88, mask: {89, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.911885318 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098273, index: 95, mask: {96, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.911909554 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098287, index: 109, mask: {110, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.911936334 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098294, index: 116, mask: {117, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.911910646 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098280, index: 102, mask: {103, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.911961842 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098301, index: 123, mask: {124, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912006567 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098267, index: 89, mask: {90, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912031393 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098281, index: 103, mask: {104, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912043626 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098288, index: 110, mask: {111, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912006597 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098274, index: 96, mask: {97, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912065047 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098295, index: 117, mask: {118, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912102527 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098302, index: 124, mask: {125, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912124168 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098260, index: 82, mask: {83, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912283699 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098265, index: 87, mask: {88, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912495969 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098258, index: 80, mask: {81, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912546204 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098272, index: 94, mask: {95, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912573034 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098279, index: 101, mask: {102, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912611837 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098286, index: 108, mask: {109, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912647715 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098293, index: 115, mask: {116, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912651532 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098300, index: 122, mask: {123, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912732505 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098257, index: 79, mask: {80, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912753194 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098299, index: 121, mask: {122, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912766218 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098271, index: 93, mask: {94, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912745589 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098264, index: 86, mask: {87, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912794742 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098278, index: 100, mask: {101, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912836521 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098285, index: 107, mask: {108, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912857180 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098292, index: 114, mask: {115, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912906863 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098270, index: 92, mask: {93, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912965213 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098256, index: 78, mask: {79, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.912996723 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098263, index: 85, mask: {86, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913038311 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098277, index: 99, mask: {100, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913066364 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098284, index: 106, mask: {107, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913102552 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098291, index: 113, mask: {114, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913142828 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098298, index: 120, mask: {121, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913212179 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098262, index: 84, mask: {85, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913400724 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098276, index: 98, mask: {99, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913450498 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098255, index: 77, mask: {78, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913495833 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098283, index: 105, mask: {106, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913536730 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098290, index: 112, mask: {113, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913558321 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098269, index: 91, mask: {92, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913595161 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098297, index: 119, mask: {120, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913611531 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098304, index: 126, mask: {127, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913897300 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098296, index: 118, mask: {119, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913897260 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098254, index: 76, mask: {77, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913900396 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098282, index: 104, mask: {105, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913902440 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098268, index: 90, mask: {91, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913902299 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098275, index: 97, mask: {98, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913898953 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098289, index: 111, mask: {112, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.914027686 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098303, index: 125, mask: {126, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:31.913899935 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2098261, index: 83, mask: {84, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:35:36.324085342 [E:onnxruntime:, inference_session.cc:2544 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/graph/graph_utils.cc:29 int onnxruntime::graph_utils::GetIndexFromName(const onnxruntime::Node&, const std::string&, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_onnx::Pow_3183for node: Mul_53/SimplifiedLayerNormFusion/ �[m Traceback (most recent call last): File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 359, in main() File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 278, in main sess = ort.InferenceSession(model_name, File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 485, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 584, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/graph/graph_utils.cc:29 int onnxruntime::graph_utils::GetIndexFromName(const onnxruntime::Node&, const std::string&, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_onnx::Pow_3183for node: Mul_53/SimplifiedLayerNormFusion/ ❌distilgpt2_fp16: ERROR - check error output�[1;31m2026-03-09 19:39:04.227355697 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178318, index: 63, mask: {64, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m�[1;31m2026-03-09 19:39:04.237909414 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178363, index: 108, mask: {109, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241899463 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178373, index: 118, mask: {119, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241915313 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178365, index: 110, mask: {111, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241945129 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178334, index: 79, mask: {80, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241957823 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178381, index: 126, mask: {127, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237910617 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178335, index: 80, mask: {81, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237916017 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178379, index: 124, mask: {125, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237915025 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178320, index: 65, mask: {66, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237912851 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178346, index: 91, mask: {92, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237914694 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178333, index: 78, mask: {79, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237915636 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178360, index: 105, mask: {106, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237914524 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178361, index: 106, mask: {107, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237916948 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178358, index: 103, mask: {104, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237923581 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178357, index: 102, mask: {103, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237923060 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178324, index: 69, mask: {70, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237925735 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178369, index: 114, mask: {115, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237923030 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178374, index: 119, mask: {120, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237923080 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178336, index: 81, mask: {82, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237935714 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178331, index: 76, mask: {77, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237929342 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178329, index: 74, mask: {75, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.245662485 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178353, index: 98, mask: {99, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.245896796 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178348, index: 93, mask: {94, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237941264 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178351, index: 96, mask: {97, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237933650 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178372, index: 117, mask: {118, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237941214 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178328, index: 73, mask: {74, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237942296 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178344, index: 89, mask: {90, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237945572 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178368, index: 113, mask: {114, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237944240 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178354, index: 99, mask: {100, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237944210 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178342, index: 87, mask: {88, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237945372 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178341, index: 86, mask: {87, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237947757 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178345, index: 90, mask: {91, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237952205 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178362, index: 107, mask: {108, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237950051 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178349, index: 94, mask: {95, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237956773 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178321, index: 66, mask: {67, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237957355 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178332, index: 77, mask: {78, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237960160 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178343, index: 88, mask: {89, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237913171 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178375, index: 120, mask: {121, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.249892105 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178370, index: 115, mask: {116, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241911977 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178339, index: 84, mask: {85, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241916566 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178322, index: 67, mask: {68, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241917878 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178327, index: 72, mask: {73, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241916736 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178340, index: 85, mask: {86, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241917988 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178377, index: 122, mask: {123, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241917698 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178356, index: 101, mask: {102, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241919231 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178350, index: 95, mask: {96, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241925623 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178378, index: 123, mask: {124, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241930722 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178366, index: 111, mask: {112, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237912280 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178337, index: 82, mask: {83, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241923068 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178376, index: 121, mask: {122, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241923048 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178355, index: 100, mask: {101, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241923619 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178338, index: 83, mask: {84, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241923018 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178323, index: 68, mask: {69, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237970259 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178352, index: 97, mask: {98, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241946873 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178371, index: 116, mask: {117, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241949227 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178380, index: 125, mask: {126, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237910667 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178347, index: 92, mask: {93, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237910647 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178319, index: 64, mask: {65, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241982600 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178330, index: 75, mask: {76, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.245904250 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178364, index: 109, mask: {110, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.245907446 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178326, index: 71, mask: {72, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.241901016 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178359, index: 104, mask: {105, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237942887 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178367, index: 112, mask: {113, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:04.237916788 [E:onnxruntime:Default, env.cc:226 ThreadMain] pthread_setaffinity_np failed for thread: 2178325, index: 70, mask: {71, }, error code: 22 error msg: Invalid argument. Specify the number of threads explicitly so the affinity is not set.�[m �[1;31m2026-03-09 19:39:05.074554374 [E:onnxruntime:, inference_session.cc:2544 operator()] Exception during initialization: /onnxruntime_src/onnxruntime/core/graph/graph_utils.cc:29 int onnxruntime::graph_utils::GetIndexFromName(const onnxruntime::Node&, const std::string&, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_onnx::Pow_1551for node: Mul_33/SimplifiedLayerNormFusion/ �[m Traceback (most recent call last): File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 359, in main() File "/src/AMDMIGraphX/tools/accuracy/accuracy_checker.py", line 278, in main sess = ort.InferenceSession(model_name, File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 485, in init self._create_inference_session(providers, provider_options, disabled_optimizers) File "/usr/local/lib/python3.10/dist-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 584, in _create_inference_session sess.initialize_session(providers, provider_options, disabled_optimizers) onnxruntime.capi.onnxruntime_pybind11_state.RuntimeException: [ONNXRuntimeError] : 6 : RUNTIME_EXCEPTION : Exception during initialization: /onnxruntime_src/onnxruntime/core/graph/graph_utils.cc:29 int onnxruntime::graph_utils::GetIndexFromName(const onnxruntime::Node&, const std::string&, bool) itr != node_args.end() was false. Attempting to get index by a name which does not exist:InsertedPrecisionFreeCast_onnx::Pow_1551for node: Mul_33/SimplifiedLayerNormFusion/ |
Member
Author
|
uh, moving back to draft so I can assess the perf drops |
bdevorem
force-pushed
the
bdevorem/AIMIGRAPHX-580-jit-implementation-for-nonzero
branch
from
4defa9d to
718b02b
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+135
to
+145
| template <index_int N, class Op, class T, class Input, class Output> | ||
| __device__ void block_scan(index idx, Op op, T init, index_int n, Input input, Output output) | ||
| { | ||
| block_scan<N>( | ||
| idx, | ||
| op, | ||
| init, | ||
| [&](auto f) -> decltype(f(index_int{})) { return idx.local_stride(n, f); }, | ||
| input, | ||
| output); | ||
| } |
There was a problem hiding this comment.
block_scan<N>(idx, ...) performs __syncthreads(), but this overload drives it via idx.local_stride(n, f) where n is runtime. When n is not an exact multiple of N (block size), different threads execute a different number of local_stride iterations, so some threads will enter block_scan while others have already exited the loop, which can deadlock/UB at the __syncthreads(). Consider rewriting this overload to iterate a fixed number of chunks (e.g., ceil_div(n, N)) that all threads execute, using an identity value for out-of-range lanes and only writing output when i < n.
Comment on lines
+47
to
+63
| block_scan<BlockSize>( | ||
| idx, | ||
| op::sum{}, | ||
| 0, | ||
| elem_num, | ||
| [&](auto j) { return float_equal(input[j], 0) ? 0 : 1; }, | ||
| [&](auto j, auto x) { | ||
| auto out_loc = x - 1; | ||
| if(float_equal(input[j], 0)) | ||
| return; | ||
|
|
||
| auto multi_idx = in_shape.multi(j); | ||
| for(index_int k = 0; k < multi_idx.size(); ++k) | ||
| { | ||
| output[k * elem_num + out_loc] = multi_idx[k]; | ||
| } | ||
| }); |
There was a problem hiding this comment.
This nonzero kernel calls block_scan<BlockSize>(..., elem_num, ...) where elem_num can be any tensor element count. With the current block_scan(..., index_int n, ...) implementation, sizes that are not a multiple of BlockSize can cause some threads to skip the final iteration and deadlock inside __syncthreads() used by the scan. Either ensure the scan implementation handles tails safely (padding/identity lanes) or constrain/guard elem_num so all threads participate in every scan iteration.
Member
Author
There was a problem hiding this comment.
solved with previous change
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
NOTE: changing the scan algorithm, perf is crazy bad
Motivation
Once again continuing on in the effort to add JIT kernels for all missing ops, I have added
nonzerohere along with thescanlibrary that is used bynonzero(and alsoprefix_scan_sumwhich is coming soon).Technical Details
The algorithm for
scanis simply ported from our device implementation; there is a potential for optimization that will be explored and implemented in a second PR. This is tracked. Our device implementation (and this JIT implementation) uses the Hillis-Steele parallel scan, which is good in terms of simplicity, but it has a single-block constraint that limits parallelism to 256 threads regardless of input size. rocPRIM has a multi-block approach that we can borrow from for better scalability.I have also removed the device implementation.
Changelog Category
Add a
CHANGELOG.mdentry for any option other thanNot Applicable