PCI passthrough causes system freeze in linux

Software & Operating Systems Linux
1

I followed the this tutorial in a attempt to setup PCI passthrough under Ubuntu. However whenever I start windows 10 under virtual machine manager my system freezes and the monitor connected to my GPU that im passing through says it is disconnected. From here I have to hard shutdown and restart my PC as nothing works except the mouse which moves around but doesn’t actually interact with anything. I updated with a custom 4.12 linux kernel from this tutorial.

Testing it with uname -r reveals this result:
4.12.8-custom

System Specs

MB_: Asus prime b350-plus (Latest bios)
CPU: Ryzen 7 1700
GPU1: XFX RX 460 (The one that will be passed through)
GPU2: Asus R7 240
RAM: 1x 8 gig Corsair 2133Mhz

Iommu Groups

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 10 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43bb] (rev 02)
IOMMU Group 10 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b7] (rev 02)
IOMMU Group 10 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b2] (rev 02)
IOMMU Group 10 1d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1d:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 10 1e:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 10 1f:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU Group 10 21:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Oland PRO [Radeon R7 240/340] [1002:6613]
IOMMU Group 10 21:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
IOMMU Group 10 24:00.0 Network controller [0280]: Intel Corporation Wireless 7260 [8086:08b1] (rev bb)
IOMMU Group 11 25:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 460] [1002:67ef] (rev cf)
IOMMU Group 11 25:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aae0]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 6 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 6 26:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 6 26:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 6 26:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]
IOMMU Group 7 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 7 27:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 7 27:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 7 27:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]
IOMMU Group 8 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 8 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 9 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 9 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 9 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 9 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 9 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 9 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 9 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 9 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]

Grub Config File

GRUB_DEFAULT=0
GRUB_HIDDEN_TIMEOUT=0
GRUB_HIDDEN_TIMEOUT_QUIET=true
GRUB_TIMEOUT=10
GRUB_DISTRIBUTOR=lsb_release -i -s 2> /dev/null || echo Debian
GRUB_CMDLINE_LINUX_DEFAULT=“radeon.blacklist=1 quiet splash amd_iommu=on pci-stub.ids=1002:67ef,1002:aae0"
GRUB_CMDLINE_LINUX=”"

NOTE: I deleted all commented lines as the formatting didn’t turn out well in the forums

/etc/libvirt/qemu/win10.xml

https://pastebin.com/g8wvnMCq

Apart from GPU passthrough my system is completely stable and is able to play games correctly. Is my system incompatible with PCI pass-through? Or have I done something horribly wrong?

2

I’m no expert, though I do at least have a working setup using intel/nvidia.

My first question is what slots are the cards in? I ask because my board has the top PCIe slot isolated, but the lower slot has random crap like network interfaces mixed in the IOMMU group, and this looks similar to yours. I don’t know if this is common, but I could not get my board to set the card in the second slot as primary display, only the first slot or integrated. So if the 460 is in the top slot your system may be using it as primary display when you try to pass it through, despite any module blacklisting or pci-stubs you have.

Second, in the VM config file, under the <devices> section, the emulator is listed as kvm-spice. On Manjaro, mine is listed as qemu-system-x86_64. You need spice for the Windows install, but that device should be removed after. Mine would not output to the graphics card until I did, only to the spice window. I don’t know why this would cause freezes though, only undesired results.

Last, have you tried setting this up with vfio-pci rather than pci-stubs? I used the archwiki guide, which I assume is adaptable to Ubuntu or whatever you’re using.

3

The 460 is on the top slot and is isolated, If I passed anything with my bottom slot I would have to pass a ton of sata, this will break more then it will fix. I will look into pci-stubs and remove the kvm spice.

I will come back when I have any new information.

4

Hi

You want to check your BIOS / UEFI settings.
Something like Above 4G and maybe Resizebar on AMD Plattforms.
Maybe have a look at that forum thread:

Or search for ACS or ACL with PCIe Passthrough.

PCI Stubbing?
Might help if the GPU driver freezes the system.
Or just put the passed through devices in the block list.

Last post might help?
https://forum.level1techs.com/t/the-pcie-passthrough-bug-is-back/