Model test failed (affable-shark's onnx weights)

[oeway]

!bioimageio test affable-shark

Getting the following issue.
Noticed that it's because of the environment is not configured correctly, is there a universal conda env which can handle not all but most of the models? Should I just use the dev env?

cc @FynnBe

See error log

``` testing affable-shark... 2024-09-12 17:15:13.365 | INFO | bioimageio.spec._internal.io_utils:open_bioimageio_yaml:112 - loading affable-shark from https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/rdf.yaml Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/rdf.yaml' to file '/home/jovyan/.cache/bioimageio/c1d5091838f82f59738702f2d8a938ed-rdf.yaml'. 100%|█████████████████████████████████████| 4.22k/4.22k [00:00<00:00, 8.48MB/s] Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/zero_mean_unit_variance.ijm' to file '/home/jovyan/.cache/bioimageio/3c123c5162a87c48333c80f7389e99a7-zero_mean_unit_variance.ijm'. 100%|█████████████████████████████████████████| 845/845 [00:00<00:00, 1.29MB/s] computing SHA256 of 3c123c5162a87c48333c80f7389e99a7-zero_mean_unit_variance.ijm Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/documentation.md' to file '/home/jovyan/.cache/bioimageio/8c032152668b4aaa68f99fb56c067631-documentation.md'. 100%|█████████████████████████████████████| 1.57k/1.57k [00:00<00:00, 3.45MB/s] SHA256 hash of downloaded file: 9b8330e6fae8403efc7a44bb7365186c14fb6222b608791baab7a4e0352dbdcd Use this value as the 'known_hash' argument of 'pooch.retrieve' to ensure that the file hasn't changed if it is downloaded again in the future. Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/test_input_0.npy' to file '/home/jovyan/.cache/bioimageio/93e503596d7d0bbf5b3937cf1a80402b-test_input_0.npy'. 100%|████████████████████████████████████████| 262k/262k [00:00<00:00, 830MB/s] computing SHA256 of 93e503596d7d0bbf5b3937cf1a80402b-test_input_0.npy (result: c Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/sample_input_0.tif' to file '/home/jovyan/.cache/bioimageio/baeb9b3639733aa6ad5f8305149a3585-sample_input_0.tif'. 100%|██████████████████████████████████████| 65.8k/65.8k [00:00<00:00, 260MB/s] computing SHA256 of baeb9b3639733aa6ad5f8305149a3585-sample_input_0.tif (result: Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/test_output_0.npy' to file '/home/jovyan/.cache/bioimageio/858a63339bdb8f80ff310e077a5079bb-test_output_0.npy'. 100%|███████████████████████████████████████| 524k/524k [00:00<00:00, 2.06GB/s] computing SHA256 of 858a63339bdb8f80ff310e077a5079bb-test_output_0.npy (result: Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/sample_output_0.tif' to file '/home/jovyan/.cache/bioimageio/d22d18e9c3592839985cac2534ac4617-sample_output_0.tif'. 100%|███████████████████████████████████████| 525k/525k [00:00<00:00, 1.51GB/s] computing SHA256 of d22d18e9c3592839985cac2534ac4617-sample_output_0.tif (result Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights.onnx' to file '/home/jovyan/.cache/bioimageio/4f6534376602b6e094e1895b93c4ee74-weights.onnx'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 370GB/s] computing SHA256 of 4f6534376602b6e094e1895b93c4ee74-weights.onnx (result: df913 Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/unet.py' to file '/home/jovyan/.cache/bioimageio/760a77af34fd5e3e1eebeff7e0669585-unet.py'. 100%|█████████████████████████████████████| 20.9k/20.9k [00:00<00:00, 41.7MB/s] computing SHA256 of 760a77af34fd5e3e1eebeff7e0669585-unet.py (result: 7f5b15948e Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights.pt' to file '/home/jovyan/.cache/bioimageio/78e469078bf5116258ceaf6f41001dd7-weights.pt'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 128GB/s] computing SHA256 of 78e469078bf5116258ceaf6f41001dd7-weights.pt (result: 608f52c Downloading data from 'https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.io/affable-shark/1.1/files/weights-torchscript.pt' to file '/home/jovyan/.cache/bioimageio/d44de80414e1fd1414b97beaaed935f5-weights-torchscript.pt'. 100%|████████████████████████████████████████| 116M/116M [00:00<00:00, 252GB/s] computing SHA256 of d44de80414e1fd1414b97beaaed935f5-weights-torchscript.pt (res 2024-09-12 17:15:46.533 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' /srv/conda/envs/notebook/lib/python3.10/site-packages/bioimageio/core/_resource_tests.py:407: UserWarning: The argument `decimal` has been depricated in favour of `relative_tolerance` and `absolute_tolerance`, with different validation logic, using `numpy.testing.assert_allclose, see 'https://numpy.org/doc/stable/reference/generated/numpy.testing.assert_allclose.html'. Passing a value for `decimal` will cause validation to revert to the old behaviour. warnings.warn( 2024-09-12 17:15:46.540 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes 2024-09-12 17:15:46.544 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' 2024-09-12 17:15:46.555 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes 2024-09-12 17:15:46.557 | INFO | bioimageio.core._resource_tests:_test_model_inference:147 - starting 'Reproduce test outputs from test inputs' 2024-09-12 17:15:46.566 | INFO | bioimageio.core._resource_tests:_test_model_inference_parametrized:242 - Testing inference with 6 different input tensor sizes

❌ bioimageio validation failed
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
source https://uk1s3.embassy.ebi.ac.uk/public-datasets/bioimage.…
format version model 0.5.3
bioimageio.spec 0.5.3.2
bioimageio.core 0.6.8

❓ location detail
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
✔️ initialized ModelDescr
to describe model 0.5.3

✔️ bioimageio.spec format
validation model 0.5.3
🔍 context.perform_io_che… True
🔍 context.root https://uk1s3.embassy.e…
🔍 context.known_files.ze… 767f2c3a50e36365c30b9e4…
🔍 context.known_files.te… c29bd6e16e3f7856217b407…
🔍 context.known_files.sa… a24b3c708b6ca6825494eb7…
🔍 context.known_files.te… 510181f38930e59e4fd8ecc…
🔍 context.known_files.sa… e8f99aabe8405427f515eba…
🔍 context.known_files.we… df913b85947f5132bcdaf81…
🔍 context.known_files.un… 7f5b15948e8e2c91f78dcff…
🔍 context.known_files.we… 608f52cd7f5119f7a7b8272…
🔍 context.known_files.we… 8410950508655a300793b38…
🔍 context.warning_level error
⚠ documentation No '# Validation'
(sub)section found in
documentation.md.

❌ Reproduce test outputs
from test inputs
❌ weights.onnx The 'onnx' model adapter
could not be created in
this environment:
ImportError(failed to
import onnxruntime: No
module named
'onnxruntime').

                       |                                                    

| | | �]8;id=839366;_resource_tests.py�_resource_tests.py�]8;;�,
line 161, in _test_model_inference with create_prediction_pipeline(
| | | |
�]8;id=688587;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=670151;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Run onnx inference
for parametrized inputs
| | ❌ | weights.onnx | The 'onnx' model
adapter could not be created in this environment: ImportError(failed to import
onnxruntime: No module named 'onnxruntime').

                       |                                                    

| | | �]8;id=227383;_resource_tests.py�_resource_tests.py�]8;;�,
line 296, in _test_model_inference_parametrized with
create_prediction_pipeline( | | |
| �]8;id=394538;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=235797;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Reproduce test
outputs from test inputs
| | ❌ | weights.pytorch_state_dict | The
'pytorch_state_dict' model adapter could not be created in this environment:
ImportError(failed to import torch: No module named 'torch').

                     |                                                      

| | | �]8;id=608868;_resource_tests.py�_resource_tests.py�]8;;�,
line 161, in _test_model_inference with create_prediction_pipeline(
| | | |
�]8;id=593174;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=396184;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Run
pytorch_state_dict inference for parametrized inputs
| | ❌ | weights.pytorch_state_dict | The
'pytorch_state_dict' model adapter could not be created in this environment:
ImportError(failed to import torch: No module named 'torch').

                     |                                                      

| | | �]8;id=13371;_resource_tests.py�_resource_tests.py�]8;;�,
line 296, in _test_model_inference_parametrized with
create_prediction_pipeline( | | |
| �]8;id=137629;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=975301;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Reproduce test
outputs from test inputs
| | ❌ | weights.torchscript | The 'torchscript'
model adapter could not be created in this environment: ImportError(failed to
import torch: No module named 'torch').

                            |                                               

| | | �]8;id=429218;_resource_tests.py�_resource_tests.py�]8;;�,
line 161, in _test_model_inference with create_prediction_pipeline(
| | | |
�]8;id=163549;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=43384;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |
| | ❌ | | Run torchscript
inference for parametrized inputs
| | ❌ | weights.torchscript | The 'torchscript'
model adapter could not be created in this environment: ImportError(failed to
import torch: No module named 'torch').

                            |                                               

| | | �]8;id=935828;_resource_tests.py�_resource_tests.py�]8;;�,
line 296, in _test_model_inference_parametrized with
create_prediction_pipeline( | | |
| �]8;id=672681;_prediction_pipeline.py�_prediction_pipeline.py�]8;;�, line 333, in create_prediction_pipeline
model_adapter = model_adapter or create_model_adapter( | | |
| �]8;id=379847;_model_adapter.py�_model_adapter.py�]8;;�, line 138, in create raise ValueError(
| | | |

</details>

[FynnBe]

this reminds me that we should probably add the conda env the test was run in (output of conda env export --from-history) to the validation summary.

Should I just use the dev env?

yeah, for most pytorch and onnx models the dev env works (any env with an up-to-date pytorch/onnxruntime package gets you very far for their great backwardcompatibility...)

there is another extra bioimageio.core[tensorflow] that install tensorflow and keras. They are not included in the current dev env.

[FynnBe]

jtlyk, affable-shark also does not pass in the current dev env, as the model's onnx weights cannot handle a batch size larger than 1

  ❌                                                                               Run onnx inference for parametrized inputs
  ❌                                                   weights.onnx                [ONNXRuntimeError] : 2 : INVALID_ARGUMENT : Got
                                                                                   invalid dimensions for input: input.1 for the
                                                                                   following indices
  index: 2 Got: 64 Expected: 256
  index: 3 Got: 64 Expected: 256
  Please fix either the inputs/outputs or the model.
                                                                                   

older core versions just didn't test with a batch size >1, so this wasn't caught before.