PyTest Migration

[Victor-Jung]

We used to write the tests directly in the GitHub Action YAML files; this is not a good practice, as it is hard to run several tests locally. PyTest is a well-known test framework that allows decoupling the testing from the GitHub CI, so users can easily run all tests for any platform locally if needed.

Currently, I use PyTest only for the Generic platform. I'd like a first round of review from @Xeratec or @diaconuccalin before moving forward. Moving the CI for other platforms will be done in a similar way.

Additionally, the PyTest suite can run tests in parallel and generate one build folder per worker to prevent conflicts.

Tagging @haugoug as he's interested in this feature.

To run all the tests of the generic platform with 8 parallel threads, you can do

pytest test_generic.py -v -n 8

You can select a specific marker with -m, like only the kernels, you can do:

pytest test_generic.py -v -n 8 -m kernels

To list all the captured tests for a given expression, you can do:

pytest test_generic.py -v -n 8 -m kernels --collect-only

To execute a specific test, you can use:

pytest 'test_generic.py::test_generic_kernels[Adder]' -v

You can also use an expression to filter tests, for instance the following command execute all test whose name contain "Adder":

pytest test_generic.py -v -k "Adder"

Added

• pytest and pytest-xdist as dependencies of Deeploy.
• A pytest.ini for the global configuration of PyTest for the project.
• conftest.py to define CLI args for PyTest for the whole project, it also defines a set of global fixtures.
• pytestRunner.py contains helper functions and fixtures for the whole project.
• test_generic.py lists the tests and sorts them into marked categories (per platform and per kernel/model).

Changed

• Drastically simplifies ci-platform-generic.yml and _runner_generic.yml to call the generic platform PyTest suite with a given marker.

Fixed

• nvidia-pyindex was broken as it now tries to build the wheel to respect the new policy on packages using pyproject. Instead of installing this package, we just add the https://pypi.ngc.nvidia.com channel to the pip config file.
• Pin versions of broken dependencies of Banshee.

PR Merge Checklist

1. The PR is rebased on the latest devel commit and pointing to devel.
2. Your PR reviewed and approved.
3. All checks are passing.
4. The CHANGELOG.md file has been updated.
5. If the docker was modified, change back its link after review.

[diaconuccalin]

Great feature, fantastically useful! :)

One more comment other than the ones I've already left, maybe add the instructions you included in the PR description to a dedicated .md file in the DeeployTest directory, with a mention of it in the big README, for better documentation for future users (including myself :) ).

[diaconuccalin]

It's only draft, but don't forget this hard-coded path :)

[diaconuccalin]

I would move these lists in a dedicated file since they're only going to get bigger in time. I would keep the current file only as a test setup file.

[Victor-Jung]

I think this makes sense, especially when I'll adapt the PR for several platforms. Then I would have a deeploytest.py (or test_suite.py) file where I declare the test functions with markers (what is currently test_generic.py) for every platform. On the side, I'll make a dedicated file per platform with the lists of test names.