Add support for libglxserver_nvidia.so

[HoKim98]

Hello, I want to use libglxserver_nvidia.so extension to create an Xorg session on my desktop.

However, the current driver image doesn't support it, while it was actually extracted in the local filesystem and could be linked.
With GPU Operator, I can find my libglxserver_nvidia.so in /run/nvidia/drivers/tmp/null/....

Just removing --x-module-path argument could resolve it.
Tested on my local k8s cluster with this patch below: quay.io/ulagbulag/openark-gpu-nvidia-driver:550.144.03-ubuntu24.04.

sh NVIDIA-Linux-$DRIVER_ARCH-$DRIVER_VERSION.run --silent \
    --ui=none \
    --no-drm \
    --no-nouveau-check \
    --no-nvidia-modprobe \
    --no-rpms \
    --no-backup \
    --no-check-for-alternate-installs \
    --no-libglx-indirect \
    --no-install-libglvnd \
    --x-prefix=/tmp/null \
    # --x-module-path=/tmp/null \
    --x-library-path=/tmp/null \
    --x-sysconfig-path=/tmp/null \
    ${install_args[@]+"${install_args[@]}"}

[skirsten]

Hi,

I have the same problem (also on gpu-operator) and I tried your solution:

ARG VERSION

FROM nvcr.io/nvidia/driver:${VERSION}

RUN sed -i '/  --x-/d' /usr/local/bin/nvidia-driver

I also tried removing all --x- overwrites except for --x-sysconfig-path like in your image.

And I can see the files inside the gpu-driver container:

$ ls /usr/lib64/xorg/modules/*
/usr/lib64/xorg/modules/drivers:
nvidia_drv.so

/usr/lib64/xorg/modules/extensions:
libglxserver_nvidia.so  libglxserver_nvidia.so.570.124.06

But they are not getting picked up by container-toolkit and not getting mounted into the app container :(
I also can't find any place to look for logs related to this.

Do you have any tips or ideas what I am doing wrong?

I checked the search path here and it should find it...

[HoKim98]

Could you restart all pods of gpu-operator and cri runtime (containerd)? Restarting the runtime class impl could help.

[skirsten]

Unfortunately that did not work. I tried it on a fresh instance so everything was using the patched version.
I also tried the nvidia-cdi and nvidia-legacy runtime class, but they are not seeing it:

Here are the logs, and the libglxserver_nvidia is just missing :( nvidia-container-toolkit-daemonset.log.

If I run ls /driver-root/usr/lib64/xorg/modules/* in the container, the files are there.

[skirsten]

Update: I was finally able to get the libglxserver_nvidia.so and nvidia_drv.so to pass through to the container. I don't know exactly which change fixed it, but I changed the Dockerfile

ARG VERSION

FROM nvcr.io/nvidia/driver:${VERSION}

RUN <<EOF
#!/bin/bash
set -eo pipefail

# Required so Xvfb does not crash (crashes for a different reason now)
sed -i '/  --no-drm/d' /usr/local/bin/nvidia-driver

# Required for graphics libraries
sed -i 's|  --x-prefix=/tmp/null|  --x-prefix=/usr|g' /usr/local/bin/nvidia-driver
sed -i 's|  --x-module-path=/tmp/null|  --x-module-path=/usr/lib/xorg/modules|g' /usr/local/bin/nvidia-driver
sed -i 's|  --x-library-path=/tmp/null|  --x-library-path=/usr/lib|g' /usr/local/bin/nvidia-driver
sed -i 's|  --x-sysconfig-path=/tmp/null|  --x-sysconfig-path=/usr/share/X11/xorg.conf.d|g' /usr/local/bin/nvidia-driver

EOF

and switched to CDI (which required a custom containerd template in k3s), and some other small changes.

In the end it didn't matter, because Xvfb was still crashing:

Reproduce

• Inside ubuntu:24.04@sha256:1e622c5f073b4f6bfad6632f2616c7f59ef256e96fe78bf6a595d1dc4376ac02 container with NVIDIA_DRIVER_CAPABILITIES=graphics,utility,compute,display,video
• This needs to be a custom docker image with this env var embedded because the part of the container-toolkit that checks for graphics can only see the env vars in the image.

$ apt-get update && apt-get install -y --no-install-recommends xvfb libglvnd0 libgl1 libglx0 libegl1 libgles2
$ Xvfb :1 -screen 0 1920x1080x24
munmap_chunk(): invalid pointer
(EE) 
(EE) Backtrace:
(EE) 0: Xvfb (OsLookupColor+0x14c) [0x5fb15539ee4c]
(EE) 1: /lib/x86_64-linux-gnu/libc.so.6 (__sigaction+0x50) [0x7663cb8c3330]
(EE) 2: /lib/x86_64-linux-gnu/libc.so.6 (pthread_kill+0x11c) [0x7663cb91cb2c]
(EE) 3: /lib/x86_64-linux-gnu/libc.so.6 (gsignal+0x1e) [0x7663cb8c327e]
(EE) 4: /lib/x86_64-linux-gnu/libc.so.6 (abort+0xdf) [0x7663cb8a68ff]
(EE) 5: /lib/x86_64-linux-gnu/libc.so.6 (perror+0xd23) [0x7663cb8a77b6]
(EE) 6: /lib/x86_64-linux-gnu/libc.so.6 (timer_settime+0x2c5) [0x7663cb926ff5]
(EE) 7: /lib/x86_64-linux-gnu/libc.so.6 (timer_settime+0x74c) [0x7663cb92747c]
(EE) 8: /lib/x86_64-linux-gnu/libc.so.6 (__libc_free+0xca) [0x7663cb92bdfa]
(EE) 9: /lib/x86_64-linux-gnu/libdrm.so.2 (drmFreeDevice+0x7d) [0x7663c82bc60d]
(EE) 10: /lib/x86_64-linux-gnu/libdrm.so.2 (drmFreeDevices+0x2e) [0x7663c82bc77e]
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 11: /lib/x86_64-linux-gnu/libnvidia-egl-gbm.so.1 (?+0x0) [0x7663b72017d0]
(EE) 12: /lib/x86_64-linux-gnu/libEGL_nvidia.so.0 (NvEglwlaf47906in+0x70460) [0x7663bdcaffd0]
(EE) 13: /lib/x86_64-linux-gnu/libEGL_nvidia.so.0 (NvEglwlaf47906in+0xd8ac) [0x7663bdc4d41c]
(EE) 14: /lib/x86_64-linux-gnu/libEGL.so.1 (eglGetError+0x425) [0x7663be186145]
(EE) unw_get_proc_name failed: no unwind info found [-10]
(EE) 15: /usr/lib/x86_64-linux-gnu/dri/swrast_dri.so (?+0x0) [0x7663cb2094a2]
(EE) 16: Xvfb (ht_dump_contents+0x8543) [0x5fb155291033]
(EE) 17: Xvfb (ht_dump_contents+0x72af) [0x5fb15528fd9f]
(EE) 18: Xvfb (_CallCallbacks+0x3c) [0x5fb15534782c]
(EE) 19: Xvfb (ht_dump_contents+0x2b0bf) [0x5fb1552b3baf]
(EE) 20: Xvfb (InitExtensions+0x89) [0x5fb155250c69]
(EE) 21: Xvfb (InitFonts+0x1e8) [0x5fb1553461a8]
(EE) 22: /lib/x86_64-linux-gnu/libc.so.6 (__libc_init_first+0x8a) [0x7663cb8a81ca]
(EE) 23: /lib/x86_64-linux-gnu/libc.so.6 (__libc_start_main+0x8b) [0x7663cb8a828b]
(EE) 24: Xvfb (_start+0x25) [0x5fb15524ed85]
(EE) 
(EE) 
Fatal server error:
(EE) Caught signal 6 (Aborted). Server aborting
(EE) 
Aborted (core dumped)

• Works fine in ubuntu:22.04 container
• Works fine if the driver is installed on the host instead

So after wasting three days of my life, I just gave up and switched to host driver. Which so far worked without any hacks 🤞

I would love if this feature could be supported in the driver container, but it seems to me it won't be as easy as hoped.