Proxmox LXC, docker GPU

unf0rg0tt3n · November 25, 2021, 5:49pm

Hi Guys,

Recently I found an old GTX 750TI laying around and I thought I could use this inside my docker setup for frigate NVR which is running in docker.

I am running proxmox 7 with an ubuntu 20.10 LXC which runs docker. I need to passthrough my 750TI to docker. So far I got the GPU inside the LXC with an nvidia-smi output but can’t run a gpu container.

nvidia-container-cli -k -d /dev/tty info (Inside LXC) output:

-- WARNING, the following logs are for debugging purposes only --

I1125 17:37:13.050875 74779 nvc.c:372] initializing library context (version=1.6.0, build=dd2c49d6699e4d8529fbeaa58ee91554977b652e)
I1125 17:37:13.051118 74779 nvc.c:346] using root /
I1125 17:37:13.051139 74779 nvc.c:347] using ldcache /etc/ld.so.cache
I1125 17:37:13.051155 74779 nvc.c:348] using unprivileged user 65534:65534
I1125 17:37:13.051196 74779 nvc.c:389] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I1125 17:37:13.051515 74779 nvc.c:391] dxcore initialization failed, continuing assuming a non-WSL environment
I1125 17:37:13.054990 74780 nvc.c:274] loading kernel module nvidia
I1125 17:37:13.055505 74780 nvc.c:278] running mknod for /dev/nvidiactl
I1125 17:37:13.055584 74780 nvc.c:282] running mknod for /dev/nvidia0
I1125 17:37:13.055641 74780 nvc.c:286] running mknod for all nvcaps in /dev/nvidia-caps
I1125 17:37:13.070350 74780 nvc.c:214] running mknod for /dev/nvidia-caps/nvidia-cap1 from /proc/driver/nvidia/capabilities/mig/config
I1125 17:37:13.070615 74780 nvc.c:214] running mknod for /dev/nvidia-caps/nvidia-cap2 from /proc/driver/nvidia/capabilities/mig/monitor
I1125 17:37:13.075360 74780 nvc.c:292] loading kernel module nvidia_uvm
I1125 17:37:13.075631 74780 nvc.c:296] running mknod for /dev/nvidia-uvm
I1125 17:37:13.075813 74780 nvc.c:301] loading kernel module nvidia_modeset
I1125 17:37:13.076056 74780 nvc.c:305] running mknod for /dev/nvidia-modeset
I1125 17:37:13.076621 74781 driver.c:101] starting driver service
I1125 17:37:13.594511 74779 nvc_info.c:758] requesting driver information with ''
I1125 17:37:13.597614 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/vdpau/libvdpau_nvidia.so.470.86
I1125 17:37:13.597894 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvoptix.so.470.86
I1125 17:37:13.598052 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.470.86
I1125 17:37:13.598176 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.470.86
I1125 17:37:13.598310 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.470.86
I1125 17:37:13.598527 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.470.86
I1125 17:37:13.598750 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.470.86
I1125 17:37:13.598887 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.470.86
I1125 17:37:13.599010 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ml.so.470.86
I1125 17:37:13.599217 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-ifr.so.470.86
I1125 17:37:13.599394 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.470.86
I1125 17:37:13.599506 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.470.86
I1125 17:37:13.599678 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.470.86
I1125 17:37:13.599815 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.470.86
I1125 17:37:13.599989 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-encode.so.470.86
I1125 17:37:13.600153 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.470.86
I1125 17:37:13.600271 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.470.86
I1125 17:37:13.600396 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.470.86
I1125 17:37:13.600589 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-cbl.so.470.86
I1125 17:37:13.600765 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.470.86
I1125 17:37:13.600948 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libnvcuvid.so.470.86
I1125 17:37:13.601341 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libcuda.so.470.86
I1125 17:37:13.601616 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.470.86
I1125 17:37:13.601807 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.470.86
I1125 17:37:13.601946 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.470.86
I1125 17:37:13.602073 74779 nvc_info.c:171] selecting /usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.470.86
W1125 17:37:13.602139 74779 nvc_info.c:397] missing library libnvidia-nscq.so
W1125 17:37:13.602157 74779 nvc_info.c:397] missing library libnvidia-fatbinaryloader.so
W1125 17:37:13.602170 74779 nvc_info.c:401] missing compat32 library libnvidia-ml.so
W1125 17:37:13.602185 74779 nvc_info.c:401] missing compat32 library libnvidia-cfg.so
W1125 17:37:13.602200 74779 nvc_info.c:401] missing compat32 library libnvidia-nscq.so
W1125 17:37:13.602222 74779 nvc_info.c:401] missing compat32 library libcuda.so
W1125 17:37:13.602237 74779 nvc_info.c:401] missing compat32 library libnvidia-opencl.so
W1125 17:37:13.602254 74779 nvc_info.c:401] missing compat32 library libnvidia-ptxjitcompiler.so
W1125 17:37:13.602272 74779 nvc_info.c:401] missing compat32 library libnvidia-fatbinaryloader.so
W1125 17:37:13.602285 74779 nvc_info.c:401] missing compat32 library libnvidia-allocator.so
W1125 17:37:13.602303 74779 nvc_info.c:401] missing compat32 library libnvidia-compiler.so
W1125 17:37:13.602320 74779 nvc_info.c:401] missing compat32 library libnvidia-ngx.so
W1125 17:37:13.602333 74779 nvc_info.c:401] missing compat32 library libvdpau_nvidia.so
W1125 17:37:13.602348 74779 nvc_info.c:401] missing compat32 library libnvidia-encode.so
W1125 17:37:13.602366 74779 nvc_info.c:401] missing compat32 library libnvidia-opticalflow.so
W1125 17:37:13.602384 74779 nvc_info.c:401] missing compat32 library libnvcuvid.so
W1125 17:37:13.602401 74779 nvc_info.c:401] missing compat32 library libnvidia-eglcore.so
W1125 17:37:13.602421 74779 nvc_info.c:401] missing compat32 library libnvidia-glcore.so
W1125 17:37:13.602443 74779 nvc_info.c:401] missing compat32 library libnvidia-tls.so
W1125 17:37:13.602458 74779 nvc_info.c:401] missing compat32 library libnvidia-glsi.so
W1125 17:37:13.602472 74779 nvc_info.c:401] missing compat32 library libnvidia-fbc.so
W1125 17:37:13.602488 74779 nvc_info.c:401] missing compat32 library libnvidia-ifr.so
W1125 17:37:13.602502 74779 nvc_info.c:401] missing compat32 library libnvidia-rtcore.so
W1125 17:37:13.602517 74779 nvc_info.c:401] missing compat32 library libnvoptix.so
W1125 17:37:13.602533 74779 nvc_info.c:401] missing compat32 library libGLX_nvidia.so
W1125 17:37:13.602550 74779 nvc_info.c:401] missing compat32 library libEGL_nvidia.so
W1125 17:37:13.602565 74779 nvc_info.c:401] missing compat32 library libGLESv2_nvidia.so
W1125 17:37:13.602579 74779 nvc_info.c:401] missing compat32 library libGLESv1_CM_nvidia.so
W1125 17:37:13.602594 74779 nvc_info.c:401] missing compat32 library libnvidia-glvkspirv.so
W1125 17:37:13.602613 74779 nvc_info.c:401] missing compat32 library libnvidia-cbl.so
I1125 17:37:13.603444 74779 nvc_info.c:297] selecting /usr/bin/nvidia-smi
I1125 17:37:13.603509 74779 nvc_info.c:297] selecting /usr/bin/nvidia-debugdump
I1125 17:37:13.603565 74779 nvc_info.c:297] selecting /usr/bin/nvidia-persistenced
I1125 17:37:13.603655 74779 nvc_info.c:297] selecting /usr/bin/nvidia-cuda-mps-control
I1125 17:37:13.603713 74779 nvc_info.c:297] selecting /usr/bin/nvidia-cuda-mps-server
W1125 17:37:13.603930 74779 nvc_info.c:423] missing binary nv-fabricmanager
I1125 17:37:13.604008 74779 nvc_info.c:341] listing firmware path /lib/firmware/nvidia/470.86
I1125 17:37:13.604087 74779 nvc_info.c:520] listing device /dev/nvidiactl
I1125 17:37:13.604107 74779 nvc_info.c:520] listing device /dev/nvidia-uvm
I1125 17:37:13.604122 74779 nvc_info.c:520] listing device /dev/nvidia-uvm-tools
I1125 17:37:13.604144 74779 nvc_info.c:520] listing device /dev/nvidia-modeset
W1125 17:37:13.604233 74779 nvc_info.c:347] missing ipc path /var/run/nvidia-persistenced/socket
W1125 17:37:13.604300 74779 nvc_info.c:347] missing ipc path /var/run/nvidia-fabricmanager/socket
W1125 17:37:13.604351 74779 nvc_info.c:347] missing ipc path /tmp/nvidia-mps
I1125 17:37:13.604373 74779 nvc_info.c:814] requesting device information with ''
I1125 17:37:13.611141 74779 nvc_info.c:705] listing device /dev/nvidia0 (GPU-e9647136-a56a-fc90-2806-54197d6dc27b at 00000000:03:00.0)
NVRM version:   470.86
CUDA version:   11.4

Device Index:   0
Device Minor:   0
Model:          NVIDIA GeForce GTX 750 Ti
Brand:          GeForce
GPU UUID:       GPU-e9647136-a56a-fc90-2806-54197d6dc27b
Bus Location:   00000000:03:00.0
Architecture:   5.0
I1125 17:37:13.611246 74779 nvc.c:423] shutting down library context
I1125 17:37:13.637276 74781 driver.c:163] terminating driver service
I1125 17:37:13.638161 74779 driver.c:203] driver service terminated successfully

Nvidia-smi output inside LXC:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.86       Driver Version: 470.86       CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:03:00.0 Off |                  N/A |
| 34%   31C    P0     1W /  38W |      0MiB /  2002MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

But when I try: docker run --rm --gpus all nvidia/cuda:11.0-base nvidia-smi

I get:

docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: container error: cgroup subsystem devices not found: unknown.

nano /etc/nvidia-container-runtime/config.toml output

disable-require = false
#swarm-resource = "DOCKER_RESOURCE_GPU"
#accept-nvidia-visible-devices-envvar-when-unprivileged = true
#accept-nvidia-visible-devices-as-volume-mounts = false

[nvidia-container-cli]
#root = "/run/nvidia/driver"
#path = "/usr/bin/nvidia-container-cli"
environment = []
#debug = "/var/log/nvidia-container-toolkit.log"
#ldcache = "/etc/ld.so.cache"
load-kmods = true
no-cgroups = false ##also tried true
#user = "root:video"
ldconfig = "@/sbin/ldconfig.real"

[nvidia-container-runtime]
debug = "/tmp/nvidia-container-runtime.log"

Does anyone have a idea to fix this?

Thanks, Dennis

modzilla · November 26, 2021, 4:42pm

Seems like docker isn’t able to create cgroups, what does your docker config json look like?

Also please post the output of docker info to see it it’s using cgroupsv2

unf0rg0tt3n · November 26, 2021, 10:57pm

Thanks for your reply,

Output of my lxc config:

lxc.apparmor.profile: unconfined
lxc.cgroup2.devices.allow: a
lxc.cap.drop: 
lxc.cgroup2.devices.allow: c 10:200 rwm
lxc.mount.entry: /dev/net dev/net none bind,create=dir
lxc.cgroup2.devices.allow: c 195:* rwm
lxc.cgroup2.devices.allow: c 507:* rwm
lxc.mount.entry: /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry: /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry: /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file

Output from ls /dev/nvidia* -l

---------- 1 root root        0 Nov 26 12:57 /dev/nvidia-modeset
crw-rw-rw- 1 root root 507,   0 Nov 24 18:31 /dev/nvidia-uvm
crw-rw-rw- 1 root root 507,   1 Nov 24 18:31 /dev/nvidia-uvm-tools
crw-rw-rw- 1 root root 195,   0 Nov 24 18:31 /dev/nvidia0
crw-rw-rw- 1 root root 195, 255 Nov 24 18:31 /dev/nvidiactl

Docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.6.3-docker)
  scan: Docker Scan (Docker Inc., v0.9.0)

Server:
 Containers: 10
  Running: 8
  Paused: 0
  Stopped: 2
 Images: 20
 Server Version: 20.10.10
 Storage Driver: fuse-overlayfs
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local local-persist
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 5b46e404f6b9f661a205e28d59c982d3634148f8
 runc version: v1.0.2-0-g52b36a2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.13.19-1-pve
 Operating System: Ubuntu 20.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 4GiB
 Name: apollo2
 ID: Y3N5:5MK7:WHFF:X7OS:PAIY:4VBP:7V3E:JZ2T:OZEA:4NUP:JTXP:37LJ
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

For some strange reason it stopped throwing errors and accepted the GPU. I could launch a GPU container. But the GPU just won’t accept the work.

modzilla · November 28, 2021, 5:52pm

github.com/NVIDIA/nvidia-docker

cgroup issue with nvidia container runtime on Debian testing

opened 08:48PM - 07 Jan 21 UTC

closed 09:46PM - 25 Jan 21 UTC

super-cooper

### 1. Issue or feature description Whenever I try to build or run an NVidia co…ntainer, Docker fails with the error message: ``` docker: Error response from daemon: OCI runtime create failed: container_linux.go:370: starting container process caused: process_linux.go:459: container init caused: Running hook #0:: error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: container error: cgroup subsystem devices not found: unknown. ``` ### 2. Steps to reproduce the issue ```bash $ docker run --rm --gpus all nvidia/cuda:11.0-base-ubuntu20.04 nvidia-smi ``` ### 3. Information to [attach](https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/) (optional if deemed irrelevant) - [x] Some nvidia-container information: `nvidia-container-cli -k -d /dev/tty info` ``` I0107 20:43:11.917241 36435 nvc.c:282] initializing library context (version=1.3.1, build=ac02636a318fe7dcc71eaeb3cc55d0c8541c1072) I0107 20:43:11.917283 36435 nvc.c:256] using root / I0107 20:43:11.917290 36435 nvc.c:257] using ldcache /etc/ld.so.cache I0107 20:43:11.917300 36435 nvc.c:258] using unprivileged user 1000:1000 I0107 20:43:11.917316 36435 nvc.c:299] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL) I0107 20:43:11.917404 36435 nvc.c:301] dxcore initialization failed, continuing assuming a non-WSL environment W0107 20:43:11.918351 36436 nvc.c:187] failed to set inheritable capabilities W0107 20:43:11.918381 36436 nvc.c:188] skipping kernel modules load due to failure I0107 20:43:11.918527 36437 driver.c:101] starting driver service I0107 20:43:11.921734 36435 nvc_info.c:680] requesting driver information with '' I0107 20:43:11.932012 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-tls.so.450.80.02 I0107 20:43:11.932402 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.450.80.02 I0107 20:43:11.932976 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ptxjitcompiler.so.450.80.02 I0107 20:43:11.933027 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-ml.so.450.80.02 I0107 20:43:11.933435 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.450.80.02 I0107 20:43:11.933470 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.450.80.02 I0107 20:43:11.933501 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.450.80.02 I0107 20:43:11.933991 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-encode.so.450.80.02 I0107 20:43:11.934024 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.450.80.02 I0107 20:43:11.934094 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libnvidia-cfg.so.450.80.02 I0107 20:43:11.934545 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libnvcuvid.so.450.80.02 I0107 20:43:11.934976 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libcuda.so.450.80.02 I0107 20:43:11.935258 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libGLX_nvidia.so.450.80.02 I0107 20:43:11.935783 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libGLESv2_nvidia.so.450.80.02 I0107 20:43:11.936188 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libGLESv1_CM_nvidia.so.450.80.02 I0107 20:43:11.936243 36435 nvc_info.c:169] selecting /usr/lib/x86_64-linux-gnu/nvidia/current/libEGL_nvidia.so.450.80.02 I0107 20:43:11.936622 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-tls.so.450.80.02 I0107 20:43:11.937013 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-glvkspirv.so.450.80.02 I0107 20:43:11.937296 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-glsi.so.450.80.02 I0107 20:43:11.937573 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-glcore.so.450.80.02 I0107 20:43:11.937881 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/libnvidia-eglcore.so.450.80.02 I0107 20:43:11.938438 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/nvidia/current/libGLX_nvidia.so.450.80.02 I0107 20:43:11.938920 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/nvidia/current/libGLESv2_nvidia.so.450.80.02 I0107 20:43:11.939282 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/nvidia/current/libGLESv1_CM_nvidia.so.450.80.02 I0107 20:43:11.939730 36435 nvc_info.c:169] selecting /usr/lib/i386-linux-gnu/nvidia/current/libEGL_nvidia.so.450.80.02 W0107 20:43:11.939751 36435 nvc_info.c:350] missing library libnvidia-opencl.so W0107 20:43:11.939756 36435 nvc_info.c:350] missing library libnvidia-fatbinaryloader.so W0107 20:43:11.939761 36435 nvc_info.c:350] missing library libnvidia-allocator.so W0107 20:43:11.939767 36435 nvc_info.c:350] missing library libnvidia-compiler.so W0107 20:43:11.939772 36435 nvc_info.c:350] missing library libnvidia-ngx.so W0107 20:43:11.939776 36435 nvc_info.c:350] missing library libvdpau_nvidia.so W0107 20:43:11.939780 36435 nvc_info.c:350] missing library libnvidia-opticalflow.so W0107 20:43:11.939785 36435 nvc_info.c:350] missing library libnvidia-fbc.so W0107 20:43:11.939790 36435 nvc_info.c:350] missing library libnvidia-ifr.so W0107 20:43:11.939795 36435 nvc_info.c:350] missing library libnvoptix.so W0107 20:43:11.939801 36435 nvc_info.c:350] missing library libnvidia-cbl.so W0107 20:43:11.939805 36435 nvc_info.c:354] missing compat32 library libnvidia-ml.so W0107 20:43:11.939810 36435 nvc_info.c:354] missing compat32 library libnvidia-cfg.so W0107 20:43:11.939814 36435 nvc_info.c:354] missing compat32 library libcuda.so W0107 20:43:11.939818 36435 nvc_info.c:354] missing compat32 library libnvidia-opencl.so W0107 20:43:11.939823 36435 nvc_info.c:354] missing compat32 library libnvidia-ptxjitcompiler.so W0107 20:43:11.939828 36435 nvc_info.c:354] missing compat32 library libnvidia-fatbinaryloader.so W0107 20:43:11.939832 36435 nvc_info.c:354] missing compat32 library libnvidia-allocator.so W0107 20:43:11.939837 36435 nvc_info.c:354] missing compat32 library libnvidia-compiler.so W0107 20:43:11.939841 36435 nvc_info.c:354] missing compat32 library libnvidia-ngx.so W0107 20:43:11.939846 36435 nvc_info.c:354] missing compat32 library libvdpau_nvidia.so W0107 20:43:11.939851 36435 nvc_info.c:354] missing compat32 library libnvidia-encode.so W0107 20:43:11.939856 36435 nvc_info.c:354] missing compat32 library libnvidia-opticalflow.so W0107 20:43:11.939860 36435 nvc_info.c:354] missing compat32 library libnvcuvid.so W0107 20:43:11.939865 36435 nvc_info.c:354] missing compat32 library libnvidia-fbc.so W0107 20:43:11.939870 36435 nvc_info.c:354] missing compat32 library libnvidia-ifr.so W0107 20:43:11.939874 36435 nvc_info.c:354] missing compat32 library libnvidia-rtcore.so W0107 20:43:11.939879 36435 nvc_info.c:354] missing compat32 library libnvoptix.so W0107 20:43:11.939884 36435 nvc_info.c:354] missing compat32 library libnvidia-cbl.so I0107 20:43:11.940108 36435 nvc_info.c:276] selecting /usr/lib/nvidia/current/nvidia-smi I0107 20:43:11.940153 36435 nvc_info.c:276] selecting /usr/lib/nvidia/current/nvidia-debugdump I0107 20:43:11.940169 36435 nvc_info.c:276] selecting /usr/bin/nvidia-persistenced W0107 20:43:11.941108 36435 nvc_info.c:376] missing binary nvidia-cuda-mps-control W0107 20:43:11.941117 36435 nvc_info.c:376] missing binary nvidia-cuda-mps-server I0107 20:43:11.941136 36435 nvc_info.c:438] listing device /dev/nvidiactl I0107 20:43:11.941142 36435 nvc_info.c:438] listing device /dev/nvidia-uvm I0107 20:43:11.941146 36435 nvc_info.c:438] listing device /dev/nvidia-uvm-tools I0107 20:43:11.941151 36435 nvc_info.c:438] listing device /dev/nvidia-modeset I0107 20:43:11.941175 36435 nvc_info.c:317] listing ipc /run/nvidia-persistenced/socket W0107 20:43:11.941193 36435 nvc_info.c:321] missing ipc /tmp/nvidia-mps I0107 20:43:11.941198 36435 nvc_info.c:745] requesting device information with '' I0107 20:43:11.947879 36435 nvc_info.c:628] listing device /dev/nvidia0 (GPU-6518be5e-14ff-e277-21aa-73b482890bee at 00000000:07:00.0) NVRM version: 450.80.02 CUDA version: 11.0 Device Index: 0 Device Minor: 0 Model: GeForce GTX 980 Ti Brand: GeForce GPU UUID: GPU-6518be5e-14ff-e277-21aa-73b482890bee Bus Location: 00000000:07:00.0 Architecture: 5.2 I0107 20:43:11.947903 36435 nvc.c:337] shutting down library context I0107 20:43:11.948696 36437 driver.c:156] terminating driver service I0107 20:43:11.949026 36435 driver.c:196] driver service terminated successfully ``` - [x] Kernel version from `uname -a` ``` Linux lambda 5.8.0-3-amd64 #1 SMP Debian 5.8.14-1 (2020-10-10) x86_64 GNU/Linux ``` - [ ] Any relevant kernel output lines from `dmesg` - [x] Driver information from `nvidia-smi -a` ``` Thu Jan 7 15:45:08 2021 +-----------------------------------------------------------------------------+ | NVIDIA-SMI 450.80.02 Driver Version: 450.80.02 CUDA Version: 11.0 | |-------------------------------+----------------------+----------------------+ | GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |===============================+======================+======================| | 0 GeForce GTX 980 Ti On | 00000000:07:00.0 On | N/A | | 0% 45C P5 29W / 250W | 403MiB / 6083MiB | 0% Default | | | | N/A | +-------------------------------+----------------------+----------------------+ +-----------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=============================================================================| | 0 N/A N/A 3023 G /usr/lib/xorg/Xorg 177MiB | | 0 N/A N/A 4833 G /usr/bin/gnome-shell 166MiB | | 0 N/A N/A 7609 G ...AAAAAAAAA= --shared-files 54MiB | +-----------------------------------------------------------------------------+ ``` - [x] Docker version from `docker version` ``` Server: Docker Engine - Community Engine: Version: 20.10.2 API version: 1.41 (minimum version 1.12) Go version: go1.13.15 Git commit: 8891c58 Built: Mon Dec 28 16:15:28 2020 OS/Arch: linux/amd64 Experimental: false containerd: Version: 1.4.3 GitCommit: 269548fa27e0089a8b8278fc4fc781d7f65a939b nvidia: Version: 1.0.0-rc92 GitCommit: ff819c7e9184c13b7c2607fe6c30ae19403a7aff docker-init: Version: 0.19.0 GitCommit: de40ad0 ``` - [x] NVIDIA packages version from `dpkg -l '*nvidia*'` _or_ `rpm -qa '*nvidia*'` ``` Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend |/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad) ||/ Name Version Architecture Description +++-======================================-==============-============-================================================================= un bumblebee-nvidia <none> <none> (no description available) ii glx-alternative-nvidia 1.2.0 amd64 allows the selection of NVIDIA as GLX provider un libegl-nvidia-legacy-390xx0 <none> <none> (no description available) un libegl-nvidia-tesla-418-0 <none> <none> (no description available) un libegl-nvidia-tesla-440-0 <none> <none> (no description available) un libegl-nvidia-tesla-450-0 <none> <none> (no description available) ii libegl-nvidia0:amd64 450.80.02-2 amd64 NVIDIA binary EGL library ii libegl-nvidia0:i386 450.80.02-2 i386 NVIDIA binary EGL library un libegl1-glvnd-nvidia <none> <none> (no description available) un libegl1-nvidia <none> <none> (no description available) un libgl1-glvnd-nvidia-glx <none> <none> (no description available) ii libgl1-nvidia-glvnd-glx:amd64 450.80.02-2 amd64 NVIDIA binary OpenGL/GLX library (GLVND variant) ii libgl1-nvidia-glvnd-glx:i386 450.80.02-2 i386 NVIDIA binary OpenGL/GLX library (GLVND variant) un libgl1-nvidia-glx <none> <none> (no description available) un libgl1-nvidia-glx-any <none> <none> (no description available) un libgl1-nvidia-glx-i386 <none> <none> (no description available) un libgl1-nvidia-legacy-390xx-glx <none> <none> (no description available) un libgl1-nvidia-tesla-418-glx <none> <none> (no description available) un libgldispatch0-nvidia <none> <none> (no description available) ii libgles-nvidia1:amd64 450.80.02-2 amd64 NVIDIA binary OpenGL|ES 1.x library ii libgles-nvidia1:i386 450.80.02-2 i386 NVIDIA binary OpenGL|ES 1.x library ii libgles-nvidia2:amd64 450.80.02-2 amd64 NVIDIA binary OpenGL|ES 2.x library ii libgles-nvidia2:i386 450.80.02-2 i386 NVIDIA binary OpenGL|ES 2.x library un libgles1-glvnd-nvidia <none> <none> (no description available) un libgles2-glvnd-nvidia <none> <none> (no description available) un libglvnd0-nvidia <none> <none> (no description available) ii libglx-nvidia0:amd64 450.80.02-2 amd64 NVIDIA binary GLX library ii libglx-nvidia0:i386 450.80.02-2 i386 NVIDIA binary GLX library un libglx0-glvnd-nvidia <none> <none> (no description available) un libnvidia-cbl <none> <none> (no description available) un libnvidia-cfg.so.1 <none> <none> (no description available) ii libnvidia-cfg1:amd64 450.80.02-2 amd64 NVIDIA binary OpenGL/GLX configuration library un libnvidia-cfg1-any <none> <none> (no description available) ii libnvidia-container-tools 1.3.1-1 amd64 NVIDIA container runtime library (command-line tools) ii libnvidia-container1:amd64 1.3.1-1 amd64 NVIDIA container runtime library ii libnvidia-eglcore:amd64 450.80.02-2 amd64 NVIDIA binary EGL core libraries ii libnvidia-eglcore:i386 450.80.02-2 i386 NVIDIA binary EGL core libraries un libnvidia-eglcore-450.80.02 <none> <none> (no description available) ii libnvidia-encode1:amd64 450.80.02-2 amd64 NVENC Video Encoding runtime library ii libnvidia-glcore:amd64 450.80.02-2 amd64 NVIDIA binary OpenGL/GLX core libraries ii libnvidia-glcore:i386 450.80.02-2 i386 NVIDIA binary OpenGL/GLX core libraries un libnvidia-glcore-450.80.02 <none> <none> (no description available) ii libnvidia-glvkspirv:amd64 450.80.02-2 amd64 NVIDIA binary Vulkan Spir-V compiler library ii libnvidia-glvkspirv:i386 450.80.02-2 i386 NVIDIA binary Vulkan Spir-V compiler library un libnvidia-glvkspirv-450.80.02 <none> <none> (no description available) un libnvidia-legacy-340xx-cfg1 <none> <none> (no description available) un libnvidia-legacy-390xx-cfg1 <none> <none> (no description available) ii libnvidia-ml-dev:amd64 11.1.1-3 amd64 NVIDIA Management Library (NVML) development files un libnvidia-ml.so.1 <none> <none> (no description available) ii libnvidia-ml1:amd64 450.80.02-2 amd64 NVIDIA Management Library (NVML) runtime library ii libnvidia-ptxjitcompiler1:amd64 450.80.02-2 amd64 NVIDIA PTX JIT Compiler ii libnvidia-rtcore:amd64 450.80.02-2 amd64 NVIDIA binary Vulkan ray tracing (rtcore) library un libnvidia-rtcore-450.80.02 <none> <none> (no description available) un libnvidia-tesla-418-cfg1 <none> <none> (no description available) un libnvidia-tesla-440-cfg1 <none> <none> (no description available) un libnvidia-tesla-450-cfg1 <none> <none> (no description available) un libnvidia-tesla-450-cuda1 <none> <none> (no description available) un libnvidia-tesla-450-ml1 <none> <none> (no description available) un libopengl0-glvnd-nvidia <none> <none> (no description available) ii nvidia-alternative 450.80.02-2 amd64 allows the selection of NVIDIA as GLX provider un nvidia-alternative--kmod-alias <none> <none> (no description available) un nvidia-alternative-legacy-173xx <none> <none> (no description available) un nvidia-alternative-legacy-71xx <none> <none> (no description available) un nvidia-alternative-legacy-96xx <none> <none> (no description available) ii nvidia-container-runtime 3.4.0-1 amd64 NVIDIA container runtime un nvidia-container-runtime-hook <none> <none> (no description available) ii nvidia-container-toolkit 1.4.0-1 amd64 NVIDIA container runtime hook ii nvidia-cuda-dev:amd64 11.1.1-3 amd64 NVIDIA CUDA development files un nvidia-cuda-doc <none> <none> (no description available) ii nvidia-cuda-gdb 11.1.1-3 amd64 NVIDIA CUDA Debugger (GDB) un nvidia-cuda-mps <none> <none> (no description available) ii nvidia-cuda-toolkit 11.1.1-3 amd64 NVIDIA CUDA development toolkit ii nvidia-cuda-toolkit-doc 11.1.1-3 all NVIDIA CUDA and OpenCL documentation un nvidia-current <none> <none> (no description available) un nvidia-current-updates <none> <none> (no description available) un nvidia-docker <none> <none> (no description available) ii nvidia-docker2 2.5.0-1 all nvidia-docker CLI wrapper ii nvidia-driver 450.80.02-2 amd64 NVIDIA metapackage un nvidia-driver-any <none> <none> (no description available) ii nvidia-driver-bin 450.80.02-2 amd64 NVIDIA driver support binaries un nvidia-driver-bin-450.80.02 <none> <none> (no description available) un nvidia-driver-binary <none> <none> (no description available) ii nvidia-driver-libs:amd64 450.80.02-2 amd64 NVIDIA metapackage (OpenGL/GLX/EGL/GLES libraries) ii nvidia-driver-libs:i386 450.80.02-2 i386 NVIDIA metapackage (OpenGL/GLX/EGL/GLES libraries) un nvidia-driver-libs-any <none> <none> (no description available) un nvidia-driver-libs-nonglvnd <none> <none> (no description available) ii nvidia-egl-common 450.80.02-2 amd64 NVIDIA binary EGL driver - common files ii nvidia-egl-icd:amd64 450.80.02-2 amd64 NVIDIA EGL installable client driver (ICD) ii nvidia-egl-icd:i386 450.80.02-2 i386 NVIDIA EGL installable client driver (ICD) un nvidia-glx-any <none> <none> (no description available) ii nvidia-installer-cleanup 20151021+12 amd64 cleanup after driver installation with the nvidia-installer un nvidia-kernel-450.80.02 <none> <none> (no description available) ii nvidia-kernel-common 20151021+12 amd64 NVIDIA binary kernel module support files ii nvidia-kernel-dkms 450.80.02-2 amd64 NVIDIA binary kernel module DKMS source un nvidia-kernel-source <none> <none> (no description available) ii nvidia-kernel-support 450.80.02-2 amd64 NVIDIA binary kernel module support files un nvidia-kernel-support--v1 <none> <none> (no description available) un nvidia-kernel-support-any <none> <none> (no description available) un nvidia-legacy-304xx-alternative <none> <none> (no description available) un nvidia-legacy-304xx-driver <none> <none> (no description available) un nvidia-legacy-340xx-alternative <none> <none> (no description available) un nvidia-legacy-340xx-vdpau-driver <none> <none> (no description available) un nvidia-legacy-390xx-vdpau-driver <none> <none> (no description available) un nvidia-legacy-390xx-vulkan-icd <none> <none> (no description available) ii nvidia-legacy-check 450.80.02-2 amd64 check for NVIDIA GPUs requiring a legacy driver un nvidia-libopencl1 <none> <none> (no description available) un nvidia-libopencl1-dev <none> <none> (no description available) ii nvidia-modprobe 460.27.04-1 amd64 utility to load NVIDIA kernel modules and create device nodes un nvidia-nonglvnd-vulkan-common <none> <none> (no description available) un nvidia-nonglvnd-vulkan-icd <none> <none> (no description available) un nvidia-opencl-dev <none> <none> (no description available) un nvidia-opencl-icd <none> <none> (no description available) un nvidia-openjdk-8-jre <none> <none> (no description available) ii nvidia-persistenced 450.57-1 amd64 daemon to maintain persistent software state in the NVIDIA driver ii nvidia-profiler 11.1.1-3 amd64 NVIDIA Profiler for CUDA and OpenCL ii nvidia-settings 450.80.02-1+b1 amd64 tool for configuring the NVIDIA graphics driver un nvidia-settings-gtk-450.80.02 <none> <none> (no description available) ii nvidia-smi 450.80.02-2 amd64 NVIDIA System Management Interface ii nvidia-support 20151021+12 amd64 NVIDIA binary graphics driver support files un nvidia-tesla-418-vdpau-driver <none> <none> (no description available) un nvidia-tesla-418-vulkan-icd <none> <none> (no description available) un nvidia-tesla-440-vdpau-driver <none> <none> (no description available) un nvidia-tesla-440-vulkan-icd <none> <none> (no description available) un nvidia-tesla-450-driver <none> <none> (no description available) un nvidia-tesla-450-vulkan-icd <none> <none> (no description available) un nvidia-tesla-alternative <none> <none> (no description available) ii nvidia-vdpau-driver:amd64 450.80.02-2 amd64 Video Decode and Presentation API for Unix - NVIDIA driver ii nvidia-visual-profiler 11.1.1-3 amd64 NVIDIA Visual Profiler for CUDA and OpenCL ii nvidia-vulkan-common 450.80.02-2 amd64 NVIDIA Vulkan driver - common files ii nvidia-vulkan-icd:amd64 450.80.02-2 amd64 NVIDIA Vulkan installable client driver (ICD) ii nvidia-vulkan-icd:i386 450.80.02-2 i386 NVIDIA Vulkan installable client driver (ICD) un nvidia-vulkan-icd-any <none> <none> (no description available) ii xserver-xorg-video-nvidia 450.80.02-2 amd64 NVIDIA binary Xorg driver un xserver-xorg-video-nvidia-any <none> <none> (no description available) un xserver-xorg-video-nvidia-legacy-304xx <none> <none> (no description available) ``` - [x] NVIDIA container library version from `nvidia-container-cli -V` ``` version: 1.3.1 build date: 2020-12-14T14:18+00:00 build revision: ac02636a318fe7dcc71eaeb3cc55d0c8541c1072 build compiler: x86_64-linux-gnu-gcc-8 8.3.0 build platform: x86_64 build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections ``` - [ ] NVIDIA container library logs (see [troubleshooting](https://github.com/NVIDIA/nvidia-docker/wiki/Troubleshooting)) - [x] Docker command, image and tag used ``` docker run --rm --gpus all nvidia/cuda:11.0-base-ubuntu20.04 nvidia-smi ```