Qemu kvm GPU passtrough not working

Operating Systems & Open Source Virtualization
,
1

Hello i am trying to do GPU passtrough to a windows VM with a seccond GPU RX 570. my main GPU is a gtx 1080TI

my idea is to have a windows VM with some games that just dont work on linux like destiny 2 and some others, at least thats the plan.

and just use steamlink to stream the games back to the host os

how ever wen i start making the VM in virt manager i get to the point of begin instalation and then i get a bunch of errors

I followd this tutorial for the GPU passtrough: (not allowd to post links?)

Unable to complete install: 'internal error: qemu unexpectedly closed the monitor: 2020-05-20T08:30:38.062876Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-register [bit 8]
2020-05-20T08:30:38.062887Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9]
2020-05-20T08:30:38.062891Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48DH).vmx-posted-intr [bit 7]
2020-05-20T08:30:38.062895Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48FH).vmx-exit-load-perf-global-ctrl [bit 12]
2020-05-20T08:30:38.062898Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(490H).vmx-entry-load-perf-global-ctrl [bit 13]
2020-05-20T08:30:38.095837Z qemu-system-x86_64: -device vfio-pci,host=0000:02:00.0,id=hostdev0,bus=pci.3,addr=0x0: vfio 0000:02:00.0: group 1 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 75, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createvm.py", line 2089, in _do_async_install
    guest.installer_instance.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 542, in start_install
    domain = self._create_guest(
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 491, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib/python3.8/site-packages/libvirt.py", line 3944, in createXML
    if ret is None:raise libvirtError('virDomainCreateXML() failed', conn=self)
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2020-05-20T08:30:38.062876Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-register [bit 8]
2020-05-20T08:30:38.062887Z qemu-system-x86_64: warning: host doesn't support requested feature: MSR(48BH).vmx-apicv-vid [bit 9]


Info about my PC

OS: Manjaro Linux x86_64 
Host: MS-7917 1.0 
Kernel: 5.6.11-1-MANJARO 
Uptime: 7 mins 
Packages: 1540 (pacman) 
Shell: bash 5.0.16 
Resolution: 5120x1440 
DE: Cinnamon 
WM: Mutter (Muffin) 
WM Theme: Mint-Y-Dark-Teal (Mint-Y-Dark-Teal) 
Theme: Mint-Y-Dark-Teal [GTK2/3] 
Icons: Papirus-Adapta-Nokto-Maia [GTK2/3] 
Terminal: gnome-terminal 
CPU: Intel i7-4790K (8) @ 4.400GHz 
GPU: NVIDIA GeForce GTX 1080 Ti 
GPU: AMD ATI Radeon RX 470/480/570/570X/580/580X 
Memory: 2346MiB / 16027MiB
2

Could you please post the output of lspci -nnvk and the pasthrough groups checker script

#!/bin/bash
shopt -s nullglob
for g in /sys/kernel/iommu_groups/*; do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;
3

output of lspci -nnvk

00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
	Subsystem: Micro-Star International Co., Ltd. [MSI] 4th Gen Core Processor DRAM Controller [1462:7917]
	Flags: bus master, fast devsel, latency 0
	Capabilities: <access denied>
	Kernel driver in use: hsw_uncore
lspci: Unable to load libkmod resources: error -12

00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 25
	Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
	I/O behind bridge: 0000e000-0000efff [size=4K]
	Memory behind bridge: f6000000-f70fffff [size=17M]
	Prefetchable memory behind bridge: 00000000c0000000-00000000d1ffffff [size=288M]
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 26
	Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
	I/O behind bridge: 0000d000-0000dfff [size=4K]
	Memory behind bridge: f7300000-f73fffff [size=1M]
	Prefetchable memory behind bridge: 00000000e0000000-00000000f01fffff [size=258M]
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller [8086:0c09] (rev 06) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 27
	Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: f7200000-f72fffff [size=1M]
	Prefetchable memory behind bridge: [disabled]
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1] (prog-if 30 [XHCI])
	Subsystem: Micro-Star International Co., Ltd. [MSI] 9 Series Chipset Family USB xHCI Controller [1462:7917]
	Flags: bus master, medium devsel, latency 0, IRQ 39
	Memory at f7400000 (64-bit, non-prefetchable) [size=64K]
	Capabilities: <access denied>
	Kernel driver in use: xhci_hcd

00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
	Subsystem: Micro-Star International Co., Ltd. [MSI] 9 Series Chipset Family ME Interface [1462:7917]
	Flags: bus master, fast devsel, latency 0, IRQ 40
	Memory at f7419000 (64-bit, non-prefetchable) [size=16]
	Capabilities: <access denied>
	Kernel driver in use: mei_me

00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad] (prog-if 20 [EHCI])
	Subsystem: Micro-Star International Co., Ltd. [MSI] 9 Series Chipset Family USB EHCI Controller [1462:7917]
	Flags: bus master, medium devsel, latency 0, IRQ 16
	Memory at f7417000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>
	Kernel driver in use: ehci-pci

00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
	Subsystem: Micro-Star International Co., Ltd. [MSI] 9 Series Chipset Family HD Audio Controller [1462:d917]
	Flags: bus master, fast devsel, latency 0, IRQ 41
	Memory at f7410000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel

00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 28
	Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: [disabled]
	Memory behind bridge: [disabled]
	Prefetchable memory behind bridge: [disabled]
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0, IRQ 29
	Bus: primary=00, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000c000-0000cfff [size=4K]
	Memory behind bridge: f7100000-f71fffff [size=1M]
	Prefetchable memory behind bridge: [disabled]
	Capabilities: <access denied>
	Kernel driver in use: pcieport

00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6] (prog-if 20 [EHCI])
	Subsystem: Micro-Star International Co., Ltd. [MSI] 9 Series Chipset Family USB EHCI Controller [1462:7917]
	Flags: bus master, medium devsel, latency 0, IRQ 23
	Memory at f7416000 (32-bit, non-prefetchable) [size=1K]
	Capabilities: <access denied>
	Kernel driver in use: ehci-pci

00:1f.0 ISA bridge [0601]: Intel Corporation Z97 Chipset LPC Controller [8086:8cc4]
	Subsystem: Micro-Star International Co., Ltd. [MSI] Z97 Chipset LPC Controller [1462:7917]
	Flags: bus master, medium devsel, latency 0
	Capabilities: <access denied>
	Kernel driver in use: lpc_ich

00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82] (prog-if 01 [AHCI 1.0])
	Subsystem: Micro-Star International Co., Ltd. [MSI] 9 Series Chipset Family SATA Controller [AHCI Mode] [1462:7917]
	Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 31
	I/O ports at f070 [size=8]
	I/O ports at f060 [size=4]
	I/O ports at f050 [size=8]
	I/O ports at f040 [size=4]
	I/O ports at f020 [size=32]
	Memory at f7415000 (32-bit, non-prefetchable) [size=2K]
	Capabilities: <access denied>
	Kernel driver in use: ahci

00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
	Subsystem: Micro-Star International Co., Ltd. [MSI] 9 Series Chipset Family SMBus Controller [1462:7917]
	Flags: medium devsel, IRQ 18
	Memory at f7414000 (64-bit, non-prefetchable) [size=256]
	I/O ports at f000 [size=32]
	Kernel driver in use: i801_smbus

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] GP102 [GeForce GTX 1080 Ti] [1462:3602]
	Flags: bus master, fast devsel, latency 0, IRQ 47
	Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at d0000000 (64-bit, prefetchable) [size=32M]
	I/O ports at e000 [size=128]
	Expansion ROM at 000c0000 [virtual] [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: nvidia

01:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] GP102 HDMI Audio Controller [1462:3602]
	Flags: bus master, fast devsel, latency 0, IRQ 17
	Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: snd_hda_intel

02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef) (prog-if 00 [VGA controller])
	Subsystem: Micro-Star International Co., Ltd. [MSI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1462:341b]
	Flags: fast devsel, IRQ 10
	Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
	Memory at f0000000 (64-bit, prefetchable) [disabled] [size=2M]
	I/O ports at d000 [disabled] [size=256]
	Memory at f7300000 (32-bit, non-prefetchable) [disabled] [size=256K]
	Expansion ROM at f7340000 [disabled] [size=128K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci

02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	Subsystem: Micro-Star International Co., Ltd. [MSI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1462:aaf0]
	Flags: bus master, fast devsel, latency 0, IRQ 5
	Memory at f7360000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci

03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804] (prog-if 02 [NVM Express])
	Subsystem: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a801]
	Flags: bus master, fast devsel, latency 0, IRQ 18, NUMA node 0
	Memory at f7200000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: nvme

05:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller [1969:e091] (rev 13)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Killer E220x Gigabit Ethernet Controller [1462:7917]
	Flags: bus master, fast devsel, latency 0, IRQ 19
	Memory at f7100000 (64-bit, non-prefetchable) [size=256K]
	I/O ports at c000 [size=128]
	Capabilities: <access denied>
	Kernel driver in use: alx

this is the output of the script

IOMMU Group 0:
	00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
IOMMU Group 1:
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
	00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
	00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller [8086:0c09] (rev 06)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
	02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev ef)
	02:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0]
	03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961 [144d:a804]
IOMMU Group 10:
	05:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E220x Gigabit Ethernet Controller [1969:e091] (rev 13)
IOMMU Group 2:
	00:14.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB xHCI Controller [8086:8cb1]
IOMMU Group 3:
	00:16.0 Communication controller [0780]: Intel Corporation 9 Series Chipset Family ME Interface #1 [8086:8cba]
IOMMU Group 4:
	00:1a.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2 [8086:8cad]
IOMMU Group 5:
	00:1b.0 Audio device [0403]: Intel Corporation 9 Series Chipset Family HD Audio Controller [8086:8ca0]
IOMMU Group 6:
	00:1c.0 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 [8086:8c90] (rev d0)
IOMMU Group 7:
	00:1c.3 PCI bridge [0604]: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 [8086:8c96] (rev d0)
IOMMU Group 8:
	00:1d.0 USB controller [0c03]: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1 [8086:8ca6]
IOMMU Group 9:
	00:1f.0 ISA bridge [0601]: Intel Corporation Z97 Chipset LPC Controller [8086:8cc4]
	00:1f.2 SATA controller [0106]: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode] [8086:8c82]
	00:1f.3 SMBus [0c05]: Intel Corporation 9 Series Chipset Family SMBus Controller [8086:8ca2]
4

The VFIO driver looks good for the RX 570, that part is correct.

That group 1 is your problem. You have to pass through every device in an IOMMU group to the same VM, or it will not work. The exception is the PCI bridges.

What you need to split the RX 570 02:00.0 and 02:00.1 into their own group, not lumped in with your NVME and 1080 ti.

You might be able to do this by playing around with settings in your motherboard’s UEFI, or by updating your UEFI. However, it is likely that you will need to patch your Linux kernel with the ACS patch to split up the groups.

5

i have looked in my bios before for anything that resembles IOMMU but i have never found a option for it or any settings with it…

how would i go about doing that patch? like i dont want to mess up kernel things by excedent (i have multiple kernals installed and also run tmeshift but still)

6

It depends on your distro.

7

i am running manjaro linux
i did find a post on reddit that says the following for manajro linux
but i think it is outdated as i am getting

archlinux-linux git repo … FAILED (unknown public key A5E9288C4FA415FA)

and the post is about a year old

1. cd
2. mkdir build
3. cd build
4. git clone https://aur.archlinux.org/linux-vfio.git
5. cd linux-vfio
6. sudo nano /etc/makepkg.conf
7. locate #MAKEFLAGS=, change to MAKEFLAGS="-j$(nproc)"
8. CTRL+O, CTRL+X
9. makepkg -si
10. sudo nano /etc/default/grub
11. add pcie_acs_override=downstream same place as enabling iommu GRUB_CMDLINE_LINUX_DEFAULT=
12. CTRL+O, CTRL+X
13. sudo grub-mkconfig -o /boot/grub/grub.cfg
14. reboot
8

That looks approximately correct. Try running gpg --recv-key A5E9288C4FA415FA and retrying.

Here is more info about the process-
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Bypassing_the_IOMMU_groups_(ACS_override_patch)
https://aur.archlinux.org/packages/linux-vfio/

9

makepkg -si is now working tthank you.

i am still not sure how to sperate the 02:00.0 and 02:00.1

but i htink this will become clear once i get there?

10

Take a look at the arch wiki page in my last post. TLDR is put pcie_acs_override=downstream as a boot parameter and reboot.

11

i did evrything and now i am getting the mssage

FAILED to start light display manager

wen ever i try to start my PC

i also removed the line
pcie_acs_override=downstream

again and the same thing now happens…

12

Try booting with the non-patched kernel.

I suspect that the Nvidia driver failed to build for the new patched kernel. See if sudo dkms autoinstall works or fails.

13

other kernal lets me boot back in to manjaro
i did sudo dkms autoinstall wen booting the broken kernal by doing ctrl+f6 loging in and runing that command. it didnt show any message no error or no succes… it did not fix it how ever (also i am runing non-free drivers for the GTX1080TI if this helps)

15

Yah, sounds like the driver dkms module did not build and install correctly for the vfio kernel.

16

is there a way to to fix this wen loading up the vfio kernal?

17

There could be a number of issues.

You need-
Linux kernel
Linux kernel headers for that exact kernel version
Nvidia drivers compatible with that exact kernel version

Then it should work.

18

alride well i dont know what Kernel headers are so i have to look that up.

and i dont know how to start about fixing the kernal also the nvidia driver.
is there a easy way to remove the non-free driver and install the open source one? maiby that could do the trick?

19

Other alternative is try moving GPUs into other slots and see if one of them is in an isolated IOMMU group, this will be board and oftentimes bios dependent

1 Like
20

They have a page about drivers here-
https://wiki.manjaro.org/Configure_Graphics_Cards

That only works if you are running on a HEDT socket, ie TR4, LGA-2011(v3), LGA 2066. The smaller sockets (AM4, LGA 115x) have only enough lanes for two slots that go directly to the CPU, so you are stuck.

21

i installed the correct kernal headers and the nonfree nvidia 440xx drivers but i still get the same error message…