Skip to content

Conversation

@ahendriksen
Copy link
Contributor

These paths were omitted before, leading to problems with more advanced use cases like Nvidia Nsight Compute.

Before

Use ncu to profile a CUDA program:

LD_LIBRARY_PATH=$(nixglhost -p) ncu  ./cuda-program

Get a failure:

==PROF== Connected to process 35030 (/home/scratch.ahendriksen_gpu/projects/2023-03-markov-clustering-raft/build/test_csroo)
==ERROR== Failed to prepare kernel for profiling                                                                  
                                                                           
==ERROR== Failed to profile "binary_search_partition_kernel" in process 35030
==PROF== Trying to shutdown target application                                                                      
==ERROR== The application returned an error code (9).                                                             
==ERROR== An error occurred while trying to profile.                                                                
==WARNING== No kernels were profiled.                                                                             
==WARNING== Profiling kernels launched by child processes requires the --target-processes all option.        

Rerun with LD_DEBUG=libs:

LD_DEBUG=libs LD_LIBRARY_PATH=$(nixglhost -p) ncu  ./cuda-program

See in the output that it cannot find libnvidia-ptxjitcompiler.so.

Check that the ptxjitciompiler so file is indeed copied to the cache:

$ fd libnvidia-ptxjitcompiler.so ~/.cache/nix-gl-host
/home/ahendriksen/.cache/nix-gl-host/229e49f198496607fb9a0a2eb54ac0f6f1e2590dced32f7aa4816a2927d48c6c/lib/libnvidia-ptxjitcompiler.so
/home/ahendriksen/.cache/nix-gl-host/229e49f198496607fb9a0a2eb54ac0f6f1e2590dced32f7aa4816a2927d48c6c/lib/libnvidia-ptxjitcompiler.so.1
/home/ahendriksen/.cache/nix-gl-host/229e49f198496607fb9a0a2eb54ac0f6f1e2590dced32f7aa4816a2927d48c6c/lib/libnvidia-ptxjitcompiler.so.525.85.12
/home/ahendriksen/.cache/nix-gl-host/3076b0246cb1468199a8444860ebaebe5ec5081f85098057a9f4d5b40c3de738/lib/libnvidia-ptxjitcompiler.so
/home/ahendriksen/.cache/nix-gl-host/3076b0246cb1468199a8444860ebaebe5ec5081f85098057a9f4d5b40c3de738/lib/libnvidia-ptxjitcompiler.so.1
/home/ahendriksen/.cache/nix-gl-host/3076b0246cb1468199a8444860ebaebe5ec5081f85098057a9f4d5b40c3de738/lib/libnvidia-ptxjitcompiler.so.525.85.12
/home/ahendriksen/.cache/nix-gl-host/3128a53abf6a5f00173c59af816b2cdfe92fef530f2e3123225ef25245d929e5/lib/libnvidia-ptxjitcompiler.so
/home/ahendriksen/.cache/nix-gl-host/3128a53abf6a5f00173c59af816b2cdfe92fef530f2e3123225ef25245d929e5/lib/libnvidia-ptxjitcompiler.so.1
/home/ahendriksen/.cache/nix-gl-host/3128a53abf6a5f00173c59af816b2cdfe92fef530f2e3123225ef25245d929e5/lib/libnvidia-ptxjitcompiler.so.525.85.12
/home/ahendriksen/.cache/nix-gl-host/a8da987559c0616b61ede2cf2a02a4a2eebd9989eb15113340224cf931b04f3f/lib/libnvidia-ptxjitcompiler.so
/home/ahendriksen/.cache/nix-gl-host/a8da987559c0616b61ede2cf2a02a4a2eebd9989eb15113340224cf931b04f3f/lib/libnvidia-ptxjitcompiler.so.1
/home/ahendriksen/.cache/nix-gl-host/a8da987559c0616b61ede2cf2a02a4a2eebd9989eb15113340224cf931b04f3f/lib/libnvidia-ptxjitcompiler.so.525.85.12

And that the nixgl-host/hash/lib/ paths are not included in the nixglhost paths:

$ nixglhost -p
/home/ahendriksen/.cache/nix-gl-host/3128a53abf6a5f00173c59af816b2cdfe92fef530f2e3123225ef25245d929e5/glx:
/home/ahendriksen/.cache/nix-gl-host/3128a53abf6a5f00173c59af816b2cdfe92fef530f2e3123225ef25245d929e5/cuda:
/home/ahendriksen/.cache/nix-gl-host/3128a53abf6a5f00173c59af816b2cdfe92fef530f2e3123225ef25245d929e5/egl:
/home/ahendriksen/.cache/nix-gl-host/a8da987559c0616b61ede2cf2a02a4a2eebd9989eb15113340224cf931b04f3f/glx:
/home/ahendriksen/.cache/nix-gl-host/a8da987559c0616b61ede2cf2a02a4a2eebd9989eb15113340224cf931b04f3f/cuda:
/home/ahendriksen/.cache/nix-gl-host/a8da987559c0616b61ede2cf2a02a4a2eebd9989eb15113340224cf931b04f3f/egl:
/home/ahendriksen/.cache/nix-gl-host/229e49f198496607fb9a0a2eb54ac0f6f1e2590dced32f7aa4816a2927d48c6c/glx:
/home/ahendriksen/.cache/nix-gl-host/229e49f198496607fb9a0a2eb54ac0f6f1e2590dced32f7aa4816a2927d48c6c/cuda:
/home/ahendriksen/.cache/nix-gl-host/229e49f198496607fb9a0a2eb54ac0f6f1e2590dced32f7aa4816a2927d48c6c/egl:
/home/ahendriksen/.cache/nix-gl-host/3076b0246cb1468199a8444860ebaebe5ec5081f85098057a9f4d5b40c3de738/glx:
/home/ahendriksen/.cache/nix-gl-host/3076b0246cb1468199a8444860ebaebe5ec5081f85098057a9f4d5b40c3de738/cuda:
/home/ahendriksen/.cache/nix-gl-host/3076b0246cb1468199a8444860ebaebe5ec5081f85098057a9f4d5b40c3de738/egl

These were omitted before, leading to problems with more advanced use
cases like Nsight Compute.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant