Return to Level1Techs.com

[elementary 0.4.1] vfio not working

#1

Hello all!

I’ve followed this guide to set up vfio, but am having trouble getting either gpu’s to use vfio-pci driver.

Specs:
Ryzen 1700
Nvidia 1070 and 1060
16GB RAM
couple of SSD’s
single monitor

Kernel (manually installed):
$ uname -a Linux zen 4.20.0-042000-generic #201812232030 SMP Mon Dec 24 01:32:58 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux

GRUB:

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet nosplash iommu=pt iommu=1 amd_iommu=on"
GRUB_CMDLINE_LINUX="bluetooth.disable_ertm=1"

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
#GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

IOMMU groups:

IOMMU Group 12 07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
IOMMU Group 12 07:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
IOMMU Group 13 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 13 09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)

I’ve first tried setting 1060 for VM, but had no luck and since there’s a bunch of other devices in group 12, I’ve tried to get 1070 to work in vfio.

lspci:

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03]
        Flags: bus master, fast devsel, latency 0, IRQ 77
        Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at d0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at d000 [size=128]
        [virtual] Expansion ROM at f5000000 [disabled] [size=512K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_415, nvidia_415_drm

07:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
        Subsystem: NVIDIA Corporation Device [10de:1c03]
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:3701]
        Flags: bus master, fast devsel, latency 0, IRQ 78
        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at f000 [size=128]
        [virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_415, nvidia_415_drm

09:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
        Subsystem: Gigabyte Technology Co., Ltd Device [1458:3701]
        Flags: bus master, fast devsel, latency 0, IRQ 11
        Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

initramfs modules:

$ cat /etc/initramfs-tools/modules
# List of modules that you want to include in your initramfs.
# They will be loaded at boot time in the order below.
#
# Syntax:  module_name [args ...]
#
# You must run update-initramfs(8) to effect this change.
#
# Examples:
#
# raid1
# sd_mod
softdep nvidia pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:1b81,10de:10f0
vfio_pci ids=10de:1b81,10de:10f0
vfio_pci
nvidia

etc/modules:

$ cat /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

vfio
vfio_iommu_type1
vfio_pci ids=10de:1b81,10de:10f0

modprobe/nvidiagpu.conf:

$ cat /etc/modprobe.d/nvidiagpu.conf 
softdep nvidia pre: vfio vfio_pci

modprobe/vfio:

$ cat /etc/modprobe.d/vfio_pci.conf
options vfio_pci ids=10de:1b81,10de:10f0

After several reboots and config checking, nvidia x server settings still shows both GPUs…

Regardless, I’ve tried creating VM, removing Display Spice device, Video QXL device and Channel Spice, and it gets stuck on “Creating domain…” part, I assume because VFIO is not working properly.

Any help would be greatly appreciated!

0 Likes