Describe the issue
ONNX Runtime tests currently generate models and test inputs dynamically inside the Python/C++ test harness. While this works well for ORT CI, it makes it difficult to reuse these test cases outside of the ORT environment.
We ran into this limitation while working on a project:
https://github.com/emmtrix/emx-onnx-cgen
emx-onnx-cgen generates C code from ONNX models, and we wanted to validate it against a broad set of real-world operator combinations and edge cases from ONNX Runtime tests. However, the current ORT test setup makes these cases hard to extract and reuse.
To address this, we built a dataset that materializes ORT test cases into standalone ONNX artifacts:
https://github.com/emmtrix/emx-ort-test-artifacts
Key idea:
- Extract
model.onnx, inputs (input_*.pb), outputs (output_*.pb)
- Preserve test semantics via metadata (
validation.json)
- Mirror the original ORT test structure
- Make everything directly consumable without rebuilding ORT
Question:
Would a reusable, versioned dataset of ORT test artifacts be useful for ONNX Runtime development or for downstream users?
Possible directions if this is interesting:
- documenting or linking such datasets from ORT
- exposing a lightweight export hook in ORT tests
- providing a maintained artifact dataset as part of the ORT project
To reproduce
Urgency
No response
Platform
Other / Unknown
OS Version
ONNX Runtime Installation
Other / Unknown
ONNX Runtime Version or Commit ID
ONNX Runtime API
Python
Architecture
X64
Execution Provider
Other / Unknown
Execution Provider Library Version
No response
Describe the issue
ONNX Runtime tests currently generate models and test inputs dynamically inside the Python/C++ test harness. While this works well for ORT CI, it makes it difficult to reuse these test cases outside of the ORT environment.
We ran into this limitation while working on a project:
https://github.com/emmtrix/emx-onnx-cgen
emx-onnx-cgengenerates C code from ONNX models, and we wanted to validate it against a broad set of real-world operator combinations and edge cases from ONNX Runtime tests. However, the current ORT test setup makes these cases hard to extract and reuse.To address this, we built a dataset that materializes ORT test cases into standalone ONNX artifacts:
https://github.com/emmtrix/emx-ort-test-artifacts
Key idea:
model.onnx, inputs (input_*.pb), outputs (output_*.pb)validation.json)Question:
Would a reusable, versioned dataset of ORT test artifacts be useful for ONNX Runtime development or for downstream users?
Possible directions if this is interesting:
To reproduce
Urgency
No response
Platform
Other / Unknown
OS Version
ONNX Runtime Installation
Other / Unknown
ONNX Runtime Version or Commit ID
ONNX Runtime API
Python
Architecture
X64
Execution Provider
Other / Unknown
Execution Provider Library Version
No response