Hello all!
I’ve followed this guide to set up vfio, but am having trouble getting either gpu’s to use vfio-pci driver.
Specs:
Ryzen 1700
Nvidia 1070 and 1060
16GB RAM
couple of SSD’s
single monitor
Kernel (manually installed):
$ uname -a Linux zen 4.20.0-042000-generic #201812232030 SMP Mon Dec 24 01:32:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
GRUB:
# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
# info -f grub -n 'Simple configuration'
GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet nosplash iommu=pt iommu=1 amd_iommu=on"
GRUB_CMDLINE_LINUX="bluetooth.disable_ertm=1"
# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"
# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console
# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480
# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true
# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"
# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"
IOMMU groups:
IOMMU Group 12 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
IOMMU Group 12 07:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 13 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 13 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
I’ve first tried setting 1060 for VM, but had no luck and since there’s a bunch of other devices in group 12, I’ve tried to get 1070 to work in vfio.
lspci:
07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) (prog-if 00 [VGA controller])
Subsystem: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03]
Flags: bus master, fast devsel, latency 0, IRQ 77
Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
Memory at c0000000 (64-bit, prefetchable) [size=256M]
Memory at d0000000 (64-bit, prefetchable) [size=32M]
I/O ports at d000 [size=128]
[virtual] Expansion ROM at f5000000 [disabled] [size=512K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_415, nvidia_415_drm
07:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:1c03]
Flags: bus master, fast devsel, latency 0, IRQ 36
Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Gigabyte Technology Co., Ltd Device [1458:3701]
Flags: bus master, fast devsel, latency 0, IRQ 78
Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
Memory at e0000000 (64-bit, prefetchable) [size=256M]
Memory at f0000000 (64-bit, prefetchable) [size=32M]
I/O ports at f000 [size=128]
[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: <access denied>
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_415, nvidia_415_drm
09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:3701]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: <access denied>
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
initramfs modules:
$ cat /etc/initramfs-tools/modules
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax: module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
softdep nvidia pre: vfio vfio_pci
vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:1b81,10de:10f0
vfio_pci ids=10de:1b81,10de:10f0
vfio_pci
nvidia
etc/modules:
$ cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
vfio
vfio_iommu_type1
vfio_pci ids=10de:1b81,10de:10f0
modprobe/nvidiagpu.conf:
$ cat /etc/modprobe.d/nvidiagpu.conf
softdep nvidia pre: vfio vfio_pci
modprobe/vfio:
$ cat /etc/modprobe.d/vfio_pci.conf
options vfio_pci ids=10de:1b81,10de:10f0
After several reboots and config checking, nvidia x server settings still shows both GPUs…
Regardless, I’ve tried creating VM, removing Display Spice device, Video QXL device and Channel Spice, and it gets stuck on “Creating domain…” part, I assume because VFIO is not working properly.
Any help would be greatly appreciated!