Use without Kubernetes

[JorritPosthuma]

Hi!

First of all, thank you for effort of enabling CUDA support on the COS images. This is the only resource I found on doing such a thing. However as our current setup has short-lived GPU processing jobs, we only use docker without kubernetes.

I managed to install the CUDA drivers using your script with the following command:

docker run -it -e "BASE_DIR=/rootfs/nvidia" \
--pid=host \
--network=host \
--privileged \
--volume=/dev:/dev \
--volume=/home/kubernetes/bin/nvidia:/rootfs/nvidia \
--volume=/etc/os-release:/rootfs/etc/os-release \
--volume=/proc/sysrq-trigger:/sysrq gcr.io/google_containers/cos-nvidia-driver-install@sha256:cb55c7971c337fece62f2bfe858662522a01e43ac9984a2dd1dd5c71487d225c

Rename sudo mv lib/ lib64 in /home/kubernetes/bin/nvidia
Run nvidia/cuda as follows:

docker run -it \
--privileged \
--volume=/home/kubernetes/bin/nvidia:/usr/local/nvidia \
nvidia/cuda \
/bin/bash

And finally inside that container:

BASE_DIR="/usr/local/nvidia"
CACHE_DIR="${BASE_DIR}/.cache"
USR_WORK_DIR="${CACHE_DIR}/usr-work"
USR_WRITABLE_DIR="${CACHE_DIR}/usr-writable"
LIB_WORK_DIR="${CACHE_DIR}/lib-work"
LIB_WRITABLE_DIR="${CACHE_DIR}/lib-writable"

mkdir -p ${USR_WRITABLE_DIR} ${USR_WORK_DIR} ${LIB_WRITABLE_DIR} ${LIB_WORK_DIR}
mount -t overlay -o lowerdir=/usr,upperdir=${USR_WRITABLE_DIR},workdir=${USR_WORK_DIR} none /usr
mount -t overlay -o lowerdir=/lib,upperdir=${LIB_WRITABLE_DIR},workdir=${LIB_WORK_DIR} none /lib # this command results in a "mount: special device none does not exist" but does work

nvidia-smi

However, I must say I'm not sure if this is the right way to go. I didn't succeed in getting nvidia-docker running on cos. Maybe you can give your ideas and opinion about the best way of running CUDA enabled docker on cos?

Thanks in advance!