GPU passthrough issue: vfio error: 0000:01:00.0: failed to open /dev/vfio/1: No such file or directory

Software & Operating Systems Linux
1

I am trying to get GPU passthrough to work on some notebooks. I have written a script that I have used before with success. Unfortunetely, I’m running into issues with the notebook I’m currently working on. It seems like a vfio binding issue to me. From what I understand I need to bind the dGPU to a vfio driver and I think that’s what might be failing here, but I could be totally wrong.

It would be great if someone of you could help me out here.

Let me give show you my script, then it’s output/error and finally some system information:

My script vm_script_clean.sh:

#!/bin/bash
SCRIPT_DIR=$(cd "$(dirname "$0")"; pwd)

RAM_SIZE=4G
CPU_CORE_COUNT=3
INSTALL_IMG=/run/media/fedora/home/windows10.iso
GPU_ROM=GTX1060.rom
DRIVE_IMG=/run/media/fedora/home/WindowsVM.img
VIRTIO_WIN_IMG=/usr/share/virtio-win/virtio-win.iso
WIN_VARS=WIN_VARS.fd
MAC_ADDRESS=52:54:BE:EF:A1:67 # generated

SMB_SHARE_FOLDER=/home/fedora/vmshare

GPU_PCI_ADDRESS=01:00.0
# If you don't use Bumblebee, you need to set the GPU_PCI_ADDRESS manually (see output of lspci) and you have to remove all following occurences of optirun in this script
# This script has only been tested with Bumblebee enabled.

GPU_IDS=$(optirun lspci -n -s "${GPU_PCI_ADDRESS}" | grep -oP "\w+:\w+" | tail -1)
GPU_VENDOR_ID=$(echo "${GPU_IDS}" | cut -d ":" -f1)
GPU_DEVICE_ID=$(echo "${GPU_IDS}" | cut -d ":" -f2)
GPU_SS_IDS=$(optirun lspci -vnn -d "${GPU_IDS}" | grep "Subsystem:" | grep -oP "\w+:\w+")
GPU_SS_VENDOR_ID=$(echo "${GPU_SS_IDS}" | cut -d ":" -f1)
GPU_SS_DEVICE_ID=$(echo "${GPU_SS_IDS}" | cut -d ":" -f2)

echo "GPU_PCI_ADDRESS: ${GPU_PCI_ADDRESS}"
echo "GPU_IDS: $GPU_IDS"
echo "GPU_VENDOR_ID: $GPU_VENDOR_ID"
echo "GPU_DEVICE_ID: $GPU_DEVICE_ID"
echo "GPU_SS_IDS: $GPU_SS_IDS"
echo "GPU_SS_VENDOR_ID: $GPU_SS_VENDOR_ID"
echo "GPU_SS_DEVICE_ID: $GPU_SS_DEVICE_ID"

# Not sure if this line is necessary
sudo echo "options vfio-pci ids=${GPU_VENDOR_ID}:${GPU_DEVICE_ID}" > /etc/modprobe.d/vfio.conf

sudo modprobe vfio-pci
sudo echo "${GPU_VENDOR_ID} ${GPU_DEVICE_ID}" > "/sys/bus/pci/drivers/vfio-pci/new_id"

# Refer https://github.com/saveriomiroddi/qemu-pinning for how to set your cpu affinity properly
sudo qemu-system-x86_64 \
  -name "Windows10-QEMU" \
  -machine type=q35,accel=kvm \
  -global ICH9-LPC.disable_s3=1 \
  -global ICH9-LPC.disable_s4=1 \
  -enable-kvm \
  -cpu host,kvm=off,hv_vapic,hv_relaxed,hv_spinlocks=0x1fff,hv_time,hv_vendor_id=12alphanum \
  -smp ${CPU_CORE_COUNT} \
  -m ${RAM_SIZE} \
  -mem-prealloc \
  -balloon none \
  -rtc clock=host,base=localtime \
  -nographic \
  -vga none \
  -serial none \
  -parallel none \
  -boot menu=on \
  -boot order=c \
  -k en-us \
  -net user,smb=${SMB_SHARE_FOLDER} \
  -spice port=5901,addr=127.0.0.1,disable-ticketing \
  -drive if=pflash,format=raw,readonly=on,file=/usr/share/OVMF/OVMF_CODE.fd \
  -drive if=pflash,format=raw,file=${WIN_VARS} \
  -drive file=${INSTALL_IMG},index=1,media=cdrom \
  -drive file=${VIRTIO_WIN_IMG},index=2,media=cdrom \
  -drive id=disk0,if=virtio,cache.direct=on,if=virtio,aio=native,format=raw,file=${DRIVE_IMG} \
  -netdev type=tap,id=net0,ifname=tap0,script=tap_ifup,downscript=tap_ifdown,vhost=on \
  -device ich9-intel-hda \
  -device hda-output \
  -device qxl,bus=pcie.0,addr=1c.4,id=video.2 \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=${GPU_PCI_ADDRESS},bus=root.1,addr=00.0,x-pci-sub-device-id=0x${GPU_SS_DEVICE_ID},x-pci-sub-vendor-id=0x${GPU_SS_VENDOR_ID},multifunction=on \
  -device pci-bridge,addr=12.0,chassis_nr=2,id=head.2 \
  -device virtio-net-pci,netdev=net0,addr=19.0,mac=${MAC_ADDRESS} \
  -device usb-tablet
  #-device usb-host,hostbus=2,hostaddr=7

# This should get executed when the vm exits
sudo echo "0000:${GPU_PCI_ADDRESS}" > "/sys/bus/pci/drivers/vfio-pci/0000:${GPU_PCI_ADDRESS}/driver/unbind"
sudo echo "OFF" >> /proc/acpi/bbswitch

The output and error the script generates:

[fedora@localhost GPU-pass-through-compatibility-check]$ sudo ./vm_script_clean.sh 
GPU_PCI_ADDRESS: 01:00.0
GPU_IDS: 10de:1c20
GPU_VENDOR_ID: 10de
GPU_DEVICE_ID: 1c20
GPU_SS_IDS: 1462:1224
GPU_SS_VENDOR_ID: 1462
GPU_SS_DEVICE_ID: 1224
qemu-system-x86_64: -balloon none: warning: This option is deprecated. Use '--device virtio-balloon' to enable the balloon device.
0+1 records in
0+1 records out
2 bytes copied, 6.7434e-05 s, 29.7 kB/s
QEMU 3.0.0 monitor - type 'help' for more information
(qemu) qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,x-pci-sub-device-id=0x1224,x-pci-sub-vendor-id=0x1462,multifunction=on: vfio error: 0000:01:00.0: failed to open /dev/vfio/1: No such file or directory
DNSMASQ terminated
0+1 records in
0+1 records out
2 bytes copied, 3.5794e-05 s, 55.9 kB/s
ip_forward disabled
./vm_script_clean.sh: line 79: /sys/bus/pci/drivers/vfio-pci/0000:01:00.0/driver/unbind: No such file or directory

I’m running Fedora 29 with kernel 4.20.6-200, bumblebee and nvidia proprietary drivers are installed.
Relevant Hardware specs:
CPU: 8th gen i7 6 cores (with iGPU)
RAM: 8GB
dGPU: GTX 1060 Mobile (non-max-Q) 6GB
vt-x and vt-d are enabled in the UEFI and IOMMU is supported by the motherboard and chipset (HM370).

lspci:

[fedora@localhost GPU-pass-through-compatibility-check]$ lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Mobile)
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:14.3 Network controller: Intel Corporation Wireless-AC 9560 [Jefferson Peak] (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:17.0 SATA controller: Intel Corporation Device a353 (rev 10)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #16 (rev f0)
00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
00:1f.0 ISA bridge: Intel Corporation Device a30d (rev 10)
00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 VGA compatible controller: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile] (rev ff)
02:00.0 Ethernet controller: Qualcomm Atheros Killer E2400 Gigabit Ethernet Controller (rev 10)

iommu groups (notice the iGPU not showing up. not sure why. also, there is a PCI bridge in the dGPU’s group. Do I have to pass/bind that as well or something? Or can I ignore it?)

[fedora@localhost GPU-pass-through-compatibility-check]$ ./lsiommu.sh 
IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers [8086:3ec4] (rev 07)
IOMMU Group 1 00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
IOMMU Group 1 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile] [10de:1c20] (rev ff)
IOMMU Group 2 00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 3 00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
IOMMU Group 3 00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 3 00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9560 [Jefferson Peak] [8086:a370] (rev 10)
IOMMU Group 4 00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 5 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:a353] (rev 10)
IOMMU Group 6 00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port #16 [8086:a337] (rev f0)
IOMMU Group 7 00:1e.0 Communication controller [0780]: Intel Corporation Device [8086:a328] (rev 10)
IOMMU Group 8 00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a30d] (rev 10)
IOMMU Group 8 00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
IOMMU Group 8 00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
IOMMU Group 8 00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU Group 9 02:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E2400 Gigabit Ethernet Controller [1969:e0a1] (rev 10)

lspci -v (the weird “!!! Unknown header type 7f” for the dGPU is caused by bumblebee,see next command)

[fedora@localhost GPU-pass-through-compatibility-check]$ lspci -v
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0
	Capabilities: <access denied>
	Kernel driver in use: skl_uncore

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 00004000-00004fff [size=4K]
	Memory behind bridge: a3000000-a40fffff [size=17M]
	Prefetchable memory behind bridge: 0000000090000000-00000000a1ffffff [size=288M]
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Mobile) (prog-if 00 [VGA controller])
	DeviceName: Onboard - Video
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 125
	Memory at a2000000 (64-bit, non-prefetchable) [size=16M]
	Memory at 80000000 (64-bit, prefetchable) [size=256M]
	I/O ports at 5000 [size=64]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: i915
	Kernel modules: i915

00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: fast devsel, IRQ 16
	Memory at a4322000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: intel_pch_thermal
	Kernel modules: intel_pch_thermal

00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) (prog-if 30 [XHCI])
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, medium devsel, latency 0, IRQ 124
	Memory at a4300000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd

00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device 7270
	Flags: fast devsel
	Memory at a431a000 (64-bit, non-prefetchable) [disabled] [size=8K]
	Memory at a4321000 (64-bit, non-prefetchable) [disabled] [size=4K]
	Capabilities: <access denied>

00:14.3 Network controller: Intel Corporation Wireless-AC 9560 [Jefferson Peak] (rev 10)
	DeviceName: Onboard - Ethernet
	Subsystem: Intel Corporation Device 0034
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at a4314000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: iwlwifi
	Kernel modules: iwlwifi

00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 126
	Memory at a4320000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: mei_me
	Kernel modules: mei_me

00:17.0 SATA controller: Intel Corporation Device a353 (rev 10) (prog-if 01 [AHCI 1.0])
	DeviceName: Onboard - SATA
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 123
	Memory at a4318000 (32-bit, non-prefetchable) [size=8K]
	Memory at a431f000 (32-bit, non-prefetchable) [size=256]
	I/O ports at 5090 [size=8]
	I/O ports at 5080 [size=4]
	I/O ports at 5060 [size=32]
	Memory at a431e000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: <access denied>
	Kernel driver in use: ahci

00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #16 (rev f0) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 122
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 00003000-00003fff [size=4K]
	Memory behind bridge: a4200000-a42fffff [size=1M]
	Prefetchable memory behind bridge: None
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 20
	[virtual] Memory at a431d000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: intel-lpss
	Kernel modules: intel_lpss_pci

00:1f.0 ISA bridge: Intel Corporation Device a30d (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, medium devsel, latency 0

00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
	DeviceName: Onboard - Sound
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 122d
	Flags: bus master, fast devsel, latency 32, IRQ 141
	Memory at a4310000 (64-bit, non-prefetchable) [size=16K]
	Memory at a4100000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: medium devsel, IRQ 16
	Memory at a431c000 (64-bit, non-prefetchable) [size=256]
	I/O ports at efa0 [size=32]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: fast devsel
	Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

01:00.0 VGA compatible controller: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile] (rev ff) (prog-if ff)
	!!! Unknown header type 7f
	Kernel modules: nouveau, nvidia_drm, nvidia

02:00.0 Ethernet controller: Qualcomm Atheros Killer E2400 Gigabit Ethernet Controller (rev 10)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 19
	Memory at a4200000 (64-bit, non-prefetchable) [size=256K]
	I/O ports at 3000 [size=128]
	Capabilities: <access denied>
	Kernel driver in use: alx
	Kernel modules: alx

optirun lspci -v (optirun is part of bumblebee to allow dGPU access)

[fedora@localhost GPU-pass-through-compatibility-check]$ optirun lspci -v
00:00.0 Host bridge: Intel Corporation 8th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0
	Capabilities: <access denied>
	Kernel driver in use: skl_uncore

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 00004000-00004fff [size=4K]
	Memory behind bridge: a3000000-a40fffff [size=17M]
	Prefetchable memory behind bridge: 0000000090000000-00000000a1ffffff [size=288M]
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Mobile) (prog-if 00 [VGA controller])
	DeviceName: Onboard - Video
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 125
	Memory at a2000000 (64-bit, non-prefetchable) [size=16M]
	Memory at 80000000 (64-bit, prefetchable) [size=256M]
	I/O ports at 5000 [size=64]
	[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: i915
	Kernel modules: i915

00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH Thermal Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: fast devsel, IRQ 16
	Memory at a4322000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: intel_pch_thermal
	Kernel modules: intel_pch_thermal

00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10) (prog-if 30 [XHCI])
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, medium devsel, latency 0, IRQ 124
	Memory at a4300000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd

00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Intel Corporation Device 7270
	Flags: fast devsel
	Memory at a431a000 (64-bit, non-prefetchable) [disabled] [size=8K]
	Memory at a4321000 (64-bit, non-prefetchable) [disabled] [size=4K]
	Capabilities: <access denied>

00:14.3 Network controller: Intel Corporation Wireless-AC 9560 [Jefferson Peak] (rev 10)
	DeviceName: Onboard - Ethernet
	Subsystem: Intel Corporation Device 0034
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at a4314000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: iwlwifi
	Kernel modules: iwlwifi

00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 126
	Memory at a4320000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: mei_me
	Kernel modules: mei_me

00:17.0 SATA controller: Intel Corporation Device a353 (rev 10) (prog-if 01 [AHCI 1.0])
	DeviceName: Onboard - SATA
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 123
	Memory at a4318000 (32-bit, non-prefetchable) [size=8K]
	Memory at a431f000 (32-bit, non-prefetchable) [size=256]
	I/O ports at 5090 [size=8]
	I/O ports at 5080 [size=4]
	I/O ports at 5060 [size=32]
	Memory at a431e000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: <access denied>
	Kernel driver in use: ahci

00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #16 (rev f0) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 122
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 00003000-00003fff [size=4K]
	Memory behind bridge: a4200000-a42fffff [size=1M]
	Prefetchable memory behind bridge: None
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1e.0 Communication controller: Intel Corporation Device a328 (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 20
	[virtual] Memory at a431d000 (64-bit, non-prefetchable) [size=4K]
	Capabilities: <access denied>
	Kernel driver in use: intel-lpss
	Kernel modules: intel_lpss_pci

00:1f.0 ISA bridge: Intel Corporation Device a30d (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, medium devsel, latency 0

00:1f.3 Audio device: Intel Corporation Cannon Lake PCH cAVS (rev 10)
	DeviceName: Onboard - Sound
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 122d
	Flags: bus master, fast devsel, latency 32, IRQ 141
	Memory at a4310000 (64-bit, non-prefetchable) [size=16K]
	Memory at a4100000 (64-bit, non-prefetchable) [size=1M]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: medium devsel, IRQ 16
	Memory at a431c000 (64-bit, non-prefetchable) [size=256]
	I/O ports at efa0 [size=32]
	Kernel driver in use: i801_smbus
	Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
	DeviceName: Onboard - Other
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: fast devsel
	Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

01:00.0 VGA compatible controller: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 147
	Memory at a3000000 (32-bit, non-prefetchable) [size=16M]
	Memory at 90000000 (64-bit, prefetchable) [size=256M]
	Memory at a0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at 4000 [size=128]
	[virtual] Expansion ROM at a4000000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

02:00.0 Ethernet controller: Qualcomm Atheros Killer E2400 Gigabit Ethernet Controller (rev 10)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Flags: bus master, fast devsel, latency 0, IRQ 19
	Memory at a4200000 (64-bit, non-prefetchable) [size=256K]
	I/O ports at 3000 [size=128]
	Capabilities: <access denied>
	Kernel driver in use: alx
	Kernel modules: alx

sudo optirun lspci -vvv -s 01:00.0 :

[fedora@localhost GPU-pass-through-compatibility-check]$ sudo optirun lspci -vv -s 01:00.0
01:00.0 VGA compatible controller: NVIDIA Corporation GP106M [GeForce GTX 1060 Mobile] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 1224
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0
	Interrupt: pin A routed to IRQ 147
	Region 0: Memory at a3000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at 90000000 (64-bit, prefetchable) [size=256M]
	Region 3: Memory at a0000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at 4000 [size=128]
	[virtual] Expansion ROM at a4000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00558  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
		DevCtl:	CorrErr- NonFatalErr- FatalErr- UnsupReq-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM+ AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s (ok), Width x16 (ok)
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR+, OBFF Via message
			 AtomicOpsCap: 32bit- 64bit- 128bitCAS-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
			 AtomicOpsCtl: ReqEn-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [250 v1] Latency Tolerance Reporting
		Max snoop latency: 0ns
		Max no snoop latency: 0ns
	Capabilities: [258 v1] L1 PM Substates
		L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
			  PortCommonModeRestoreTime=255us PortTPowerOnTime=10us
		L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
			   T_CommonMode=0us LTR1.2_Threshold=0ns
		L1SubCtl2: T_PwrOn=10us
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [420 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr-
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- AdvNonFatalErr+
		AERCap:	First Error Pointer: 00, ECRCGenCap- ECRCGenEn- ECRCChkCap- ECRCChkEn-
			MultHdrRecCap- MultHdrRecEn- TLPPfxPres- HdrLogCap-
		HeaderLog: 00000000 00000000 00000000 00000000
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] Secondary PCI Express <?>
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

Kernel parameters:

[fedora@localhost GPU-pass-through-compatibility-check]$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-4.20.6-200.fc29.x86_64 root=/dev/mapper/fedora-root ro pcie_acs_override=downstream intel_iommu=on,igfx_off nouveau.modeset=0 rd.driver.blacklist=nouveau modprobe.blacklist=nouveau nvidia-drm.modeset=1 rd.driver.pre=vfio-pci amd_iommu=on iommu=1 resume=/dev/mapper/fedora-swap rd.lvm.lv=fedora/root rd.lvm.lv=fedora/swap rhgb quiet LANG=en_US.UTF-8

lsmod | grep vfio :

[fedora@localhost GPU-pass-through-compatibility-check]$ lsmod | grep vfio
vfio_pci               61440  0
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       32768  0
vfio                   36864  3 kvmgt,vfio_iommu_type1,vfio_pci
irqbypass              16384  2 vfio_pci,kvm
2

To be honest I am no expert on GPU passthrough and I haven’t done it once jet. But it seems Bumblebee kicks in before VIFO does.
Maybe you need to remove nvidia-drm.modeset at boot. Or change the orders of the rd.hooks at boot, or change module load order.

You can also add the GPU ID into boot via a rd.hook.

sorry in advanced, if this wont work. As said I haven’t done a GPU passthrough jet.

But I am quite fascinated at the project you created.

3

modeset on nvidia shouldn’t be happening so early. Usually NVIDIA modeset happens when an Xserver is called late in the boot process. If it’s any earlier, modeset in a kernel parameter overrides VFIO.

4

Thanks for your replies. I removed nvidia-drm.modeset=1 and add modified modprobe.blacklist=nouveau to modprobe.blacklist=nouveau,nvidia,nvidia-drm,nvidia-updates-,nvidia-experimental, but the result is the same. :confused:

Edit:
I also tried changing rd.driver.blacklist=nouveau to rd.driver.blacklist=nouveau,nvidia.

But lspci is still saying

Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia

@FurryJackman you mention Xserver, I’m not sure if that’s a thing on my system since Fedora 29 comes with Wayland from what I understand.

One more thing:
I am planning to switch between using the GPU in the vm and on the host without reboots. So I will defintiely need the nvidia driver at some point.
Shouldn’t it be totally okay for the nvidia driver to be loaded at boot?

I was thinking that I was simply binding the vfio driver using:

sudo echo "${GPU_VENDOR_ID} ${GPU_DEVICE_ID}" > "/sys/bus/pci/drivers/vfio-pci/new_id"

and unbinding it again using:

sudo echo "0000:${GPU_PCI_ADDRESS}" > "/sys/bus/pci/drivers/vfio-pci/0000:${GPU_PCI_ADDRESS}/driver/unbind"

I may have solved it by explicitly unbinding the nvidia driver first:

echo "0000:${GPU_PCI_ADDRESS}" > "/sys/bus/pci/devices/0000:${GPU_PCI_ADDRESS}/driver/unbind"

I’ll report back when I know more.

Edit:
Okay, got it, that was the problem. :slight_smile:

5

It could be that the APU is deactivated via BIOS. You can check the boot process with some like dmesg -T | grep -iE "vga|3d|nvidia|vfio". Maybe you can get a hint what is going wrong.

Addition …
Ooops, I didn’t notice you have edited your post.
Lucky you. Have fun with your script.

6

Well you can see the iGPU in my lspci so that was out of question to me :wink:

00:02.0 VGA compatible controller: Intel Corporation UHD Graphics 630 (Mobile)

I will and I will even publish my scripts on GitHub once I got everything automated to a very noob-friendly point.

7

What is the status of this?

I do have one question for vfio blocking, what is the reason for blocking modules and not PCI IDs? Is it due to the IDs changing?

8

Basically because it works. But I’m sure there are other solutions that work.
Are you talking about the kernel parameters to blacklist the nouveau driver? I did that to ensure that the proprietary Nvidia driver is used. It sounds to me like your suggestion would be to completely disable the GPU on boot? I actually don’t want that because I’m using the GPU on the host machine while the VM is not running.
And yeah if I would use PCI IDs in the kernel parameters (if that’s how this would work), I wouldn’t like that because I need a portable solution that runs on a USB stick and can be booted on any notebook. And since all that code you saw above is the predecessor of this project, it really has to be as easy as possible.

9

Sorry to bother, could I have your vbios? I need your exact vbios and I messed up my only backup

10

@sondreavik
You can’t just use any vbios. You have to use one that fits your machine. I uploaded the vbioses of all models that I tested to https://www.techpowerup.com/vgabios/
Although my vbioses are most likely in the Unverified Uploads category.