Further modernization of the build backend to reduce `setup.py` usage

[leofang]

Creating an issue to capture the internal discussions.

Currently both cuda.bindings and cuda.core use setuptools as the build backend and have a mixture of pyproject.toml static declarations and setup.py dynamic logics. We have two choices here:

• Keep setuptools but further reduce the amount of code in setup.py:
  • This seems to become possible with recent setuptools, where Cython extensions can now be declared in pyproject.toml.
• Switch to use scikit-build-core:
  • This has the benefit of reusing CUDA discovery mechanism (FindCUDAToolkit) and other nice build system features provided by CMake (see the relevant discussion in Splayed layout support #46).
  • The potential downside is to maintain a custom CMakeLists.txt which is arguably a burden for non-C++ (=C & Python) oriented developers. Our Cython extensions are very simple to build.
    • However, this might become necessary once we start integrating some features from cudax::, e.g. Build MemoryResource on top of cudax::any_resource #528.

[leofang]

cc @ZzEeKkAa for vis (since cuda.bindings's setup.py has a chunk of code copied from nvmath-python and other Python projects)

[mdboom]

Unless I'm missing a way to achieve this today, another nice benefit of this change would be parallel builds (utilizing all cores).

[leofang]

Setting CUDA_PYTHON_PARSER_CACHING=1 and CUDA_PYTHON_PARALLEL_LEVEL=$(nproc) will make building cuda-bindings reasonably faster!

https://nvidia.github.io/cuda-python/cuda-bindings/latest/environment_variables.html

[ZzEeKkAa]

I guess it is better to revisit this, once #661 is implemented. We need headers path, while building extensions

[cryos]

I guess it is better to revisit this, once #661 is implemented. We need headers path, while building extensions

Not really as once we move to scikit-build-core CMake would find the libraries and headers. You could add prefixes, but at least from my perspective a lot of the motivation was to benefit from all the capabilities that are already present in CMake.

[ZzEeKkAa]

Is CMake aware of different locations of ctk (venv, conda, sysctk) ? Correct CTK location still need to be passed there