[FEA]: Ability to provide an extra path for cuda-pathfinder to search with high priority

[benhg]

Is this a duplicate?

• I confirmed there appear to be no duplicate issues for this request and that I agree to the Code of Conduct

Area

cuda.pathfinder

Is your feature request related to a problem? Please describe.

The cuda-pathfinder's search path prioritizes libraries from wheels. For example, libnvshmem_host.so comes from libnvidia-nvshmem-cuXX wheel. If I'm working from a development branch of nvshmem, I may need to manually copy libraries over the site-packages directory in order to get my custom libraries to be opened. LD_LIBRARY_PATH is much lower in the priority order (on purpose), so that won't work either.

Describe the solution you'd like

A few different ideas come to mind:

• <LIBNAME_UPPER_CASE>_PATH_OVERRIDE so each library has an ability to be overridden independently
• CUDA_PATHFINDER_TOP_PRIORITY_PATH which applies to all pathfinder searches
• Adding a <library>_HOME environment variable, and prioritizing it above other search paths

Describe alternatives you've considered

I think I listed all 3 above. The current hacky WAR is to copy .so files around on the system.

Additional context

No response

[leofang]

Why don't you just remove the wheel from the Python env, and let the system search kick in (through LD_LIBRARY_PATH)? This is not a typical request especially for library teams who own both C and Python libraries. If you don't keep the hygiene of your dev env, at one point things will just break and we can't be blamed.

[benhg]

Why don't you just remove the wheel from the Python env, and let the system search kick in (through LD_LIBRARY_PATH)? This is not a typical request especially for library teams who own both C and Python libraries. If you don't keep the hygiene of your dev env, at one point things will just break and we can't be blamed.

We could certainly do that. It's just one extra step to make sure pip install never pulls back in those dependencies. I agree that it is very much a niche/developer-only feature and may not be worth doing if it only serves a small value.

[leofang]

pip install --no-deps ... will help you avoid installing dependencies. Alternatively, you can organize non-nvSHMEM dependencies into a, say,requirements-dev-core.txt file, and then pip install -r requirements-dev-core.txt to just avoid nvSHMEM.

I'd prefer us to not commit to any overwrite mechanisms for now (preferably forever). I don't want to repeat the nightmare of having RPATH vs RUNPATH and having to worry about the search precedence. The less factors participating in our search logic, the better.

[benhg]

Understandable. This is not a dealbreaker for us (or probably anyone else) if we can't have it.

[kkraus14]

I agree we shouldn't reorder standard paths, but I agree with @benhg that there's value in having an "escape hatch" of sorts to override pathfinder behavior. This will inevitably come in handy when someone needs to pull in a debug build of a library or something else. I think if we have pathfinder specific environment variables that allow controlling on a per library and per component (library vs headers vs executables vs etc.) basis that it would be a valuable addition.