When setting clock frequencies using NVML during tuning, we'd like Kernel Tuner to reset the GPU applications clock when it is done, also when the process terminates for some reason.
When relying on the nvidia-smi-fallback option this requires launching a subprocess, but that fails if we are halfway through a crash:
Exception ignored in: <function nvml.__del__ at 0x153d40bc2ac0>
Traceback (most recent call last):
File "/home/bwn200/kernel_tuner/kernel_tuner/observers/nvml.py", line 88, in __del__
File "/home/bwn200/kernel_tuner/kernel_tuner/observers/nvml.py", line 219, in reset_clocks
File "/home/bwn200/kernel_tuner/kernel_tuner/observers/nvml.py", line 183, in set_clocks
File "/home/bwn200/miniconda3/lib/python3.11/subprocess.py", line 548, in run
File "/home/bwn200/miniconda3/lib/python3.11/subprocess.py", line 1026, in __init__
File "/home/bwn200/miniconda3/lib/python3.11/subprocess.py", line 1882, in _execute_child
File "<frozen os>", line 629, in get_exec_path
ImportError: sys.meta_path is None, Python is likely shutting down
This can likely be fixed by using atexit instead of the destructor of the nvml wrapper.
When setting clock frequencies using NVML during tuning, we'd like Kernel Tuner to reset the GPU applications clock when it is done, also when the process terminates for some reason.
When relying on the nvidia-smi-fallback option this requires launching a subprocess, but that fails if we are halfway through a crash:
This can likely be fixed by using atexit instead of the destructor of the nvml wrapper.