I put together a new home server using old threadripper parts and an interesting X399 board from Asrock that included dual 10g networking and an IPMI equivalent. Hardware:
- TR 1920x
- ASRock Rack X399D8A-2T
- Seasonic 800w focus plus
- 8xSAS drives ( Seagate Constellation E3 3TB)
- 2xSamsung 970 Evo plus nvme m.2 (mirror 500GB )
- 1xSamsung 860 Evo SATA ( Cache )
- LSI SAS 9210-8i
Installed Proxmox, created a TrueNas Core VM, passthrough the LSI controller and the small SSD created another zfs pool over there and created network shares to communicate, so far so good.
Then I followed “Craft Computing Youtube Guide on This” IDK if links are permitted here, so I will not link, but searching for plex on proxmox is likely to find you his video. I followed these steps.
- Created Ubuntu server LTS 20 VM
- Enabled repository updating as per Plex Guide
- Installed Plex
- Added libraries from the share
— GPU pass trough Journey begins here —
ON HOST (Proxmox)
- Included “quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction nofb nomodeset video=vesafb:off,efifb:off” /etc/kernel/cmdline ( I use systemd not grub )
- proxmox-boot-tool refresh
- included the following in /etc/modules:
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd - IOMMU interrupt remaping:
echo “options vfio_iommu_type1 allow_unsafe_interrupts=1” > /etc/modprobe.d/iommu_unsafe_interrupts.conf
echo “options kvm ignore_msrs=1” > /etc/modprobe.d/kvm.conf - Blacklist drivers
echo “blacklist radeon” >> /etc/modprobe.d/blacklist.conf
echo “blacklist nouveau” >> /etc/modprobe.d/blacklist.conf
echo “blacklist nvidia” >> /etc/modprobe.d/blacklist.conf - reboot
- added GPU to VFIO ( I made sure the I got the correct Id’s)
echo “options vfio-pci ids=10de:1c31,10de:10f1 disable_vga=1”> /etc/modprobe.d/vfio.conf - update-initramfs -u
-reboot
ON GUEST ( Ubuntu LTS 20.04.2 live server amd64 )
-
Download nVidia Drivers
sudo apt update
wget https : // international.download.nvidia.com /XFree86/Linux-x86_64 /###.##.##/NVIDIA-Linux-x86_64- ###.##.## . run
sudo chmod +x NVIDIA-Linux-x86_64-###.##.##.run -
Disable Nouveau drivers in kernel-
sudo bash -c “echo blacklist nouveau > /etc/modprobe.d/blacklist-nvidia-nouveau.conf”
sudo bash -c “echo options nouveau modset=0 >> /etc/modprobe.d/blacklist-nvidia-nouveau.conf”
sudo update-initramfs -u -
reboot
ON HOST (PROXMOX)
- Hide VM identifiers from Nvidia
SSH into ProxMox
cd /etc/pve/qemu-server
nano ###.conf (# is the VM identifier of your Plex server)
Modify cpu line…
cpu: host,hidden=1
Save file and exit.
Start Plex VM.
Nvidia-smi to confirm GPU is working.
—THE PROBLEM—
At this lspci -v inside the VM returns the PCIe device and returns the correct driver in use, but…
nvidia-msi
No devices Found
I’ve also changed the bios to UEFI ( OMVF )
and turned off SR-IOV on bare metal bios, since I read this was a good idea because I’m passing through a single GPU
At this point I don’t know if nvdia-msi is a good testing tool or if there are better ways to make sure that it is working
If there are other commands or config files I should look like pls let me know and how to test if the gpu is working or not.