[Question]: Unable to deploy gpu operator in RKE2 and CRIO as container runtime

[Aashiqps]

hi, i have add a gpu worker node in rke2 cluster, crio is used as container runtime. i have manually installed the driver and container toolkit and also done the container toolkit config for crio. the document in rke2 only shows about containerd , so i refered the gpu-operator helm document for the installation but iam getting below error, i installed with driver and toolkit disabled in the values.yaml. kindly provide a guidance for rke2 and crio.

error
Failed to create pod sandbox: rpc error: code = Unknown desc = failed to find runtime handler nvidia from runtime list map[crun:0xc000305680 runc:0xc000305980]

steps i did for installing container toolkit

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.19.0-1
  sudo apt-get install -y \
      nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
      libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

sudo nvidia-ctk runtime configure --runtime=crio
sudo systemctl restart crio

Output

root@k8s-worker-gpu-01:/etc/crio/conf.d# cat 99-nvidia.toml 

[crio]

  [crio.runtime]

    [crio.runtime.runtimes]

      [crio.runtime.runtimes.nvidia]
        runtime_path = "/usr/bin/nvidia-container-runtime"
        runtime_type = "oci"

Fri Mar 20 07:39:02 2026       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 590.48.01              Driver Version: 590.48.01      CUDA Version: 13.1     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Quadro RTX 5000                Off |   00000000:37:00.0 Off |                  Off |
| 33%   36C    P0             45W /  230W |       0MiB /  16384MiB |      2%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  Quadro RTX 5000                Off |   00000000:86:00.0 Off |                  Off |
| 34%   35C    P0             48W /  230W |       0MiB /  16384MiB |      5%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+