PCI Pass Through IOMMU Groups Help

Operating Systems & Open Source Linux
1

Ubuntu 18.04.2
Kernel - 5.0.7-050007-generic

Virt manager error:
Error starting domain: internal error: process exited while connecting to monitor: 2019-04-11T00:23:06.875147Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev1,bus=pci.7,addr=0x0: vfio error: 0000:02:00.0: group 1 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Traceback (most recent call last):
File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 89, in cb_wrapper
callback(asyncjob, *args, **kwargs)
File “/usr/share/virt-manager/virtManager/asyncjob.py”, line 125, in tmpcb
callback(*args, **kwargs)
File “/usr/share/virt-manager/virtManager/libvirtobject.py”, line 82, in newfn
ret = fn(self, *args, **kwargs)
File “/usr/share/virt-manager/virtManager/domain.py”, line 1508, in startup
self._backend.create()
File “/usr/lib/python2.7/dist-packages/libvirt.py”, line 1062, in create
if ret == -1: raise libvirtError (‘virDomainCreate() failed’, dom=self)
libvirtError: internal error: process exited while connecting to monitor: 2019-04-11T00:23:06.875147Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,id=hostdev1,bus=pci.7,addr=0x0: vfio error: 0000:02:00.0: group 1 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

IOMMU groups:
/sys/kernel/iommu_groups/7/devices/0000:00:1b.0
/sys/kernel/iommu_groups/15/devices/0000:06:00.0
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/13/devices/0000:03:00.0
/sys/kernel/iommu_groups/3/devices/0000:00:08.0
/sys/kernel/iommu_groups/11/devices/0000:00:1d.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.2
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.1
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.3
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/16/devices/0000:07:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/14/devices/0000:05:00.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/12/devices/0000:00:1f.3
/sys/kernel/iommu_groups/12/devices/0000:00:1f.4
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:1c.4
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1c.3

$ lspci -vnn | grep -iP “vga|amdgpu|nvidia|nouveau|vfio-pci”

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1f08] (rev a1) (prog-if 00 [VGA controller])
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f9] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ada] (rev a1) (prog-if 30 [XHCI])
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1adb] (rev a1)
Kernel driver in use: nvidia-gpu
Kernel modules: i2c_nvidia_gpu
02:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X] [1002:67df] (rev ef) (prog-if 00 [VGA controller])
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
Kernel driver in use: vfio-pci

$lspci -k
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X] (rev ef)
Subsystem: ASUSTeK Computer Inc. Ellesmere [Radeon RX 470/480/570/580]
Kernel driver in use: vfio-pci
Kernel modules: amdgpu
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580]
Subsystem: ASUSTeK Computer Inc. Ellesmere [Radeon RX 580]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

I think what I need to do is to put GPU, Audio into their own group.

I need to put 0000:02:00.0 and 0000:02:00.1 into their own groups. Can I use the ACS patch from (https://drive.google.com/file/d/0B7qwHfGFYCIIcGxEcHc5X3l0R1k/view | Play games in Windows on Linux! PCI passthrough quick guide)
or do I need to change my kernel to 4.7, or is there a patch one for a later kernel? or is it a completely different issue?

Thanks

2

You are correct that the problem is that the GPU+GPU audio are in the same group as other things. You have a couple options for your type of situation-

  1. Pass everything in the group through.(does not work for you)
  2. Switch around cards in slots on the mobo(again does not work for you since all the slots are in one group)
  3. Buy a better motherboard(last resort)
  4. Update your UEFI(may work, who knows)
  5. ACS patch as mentioned above

Here are instructions for applying the ACS patch on Ubuntu systems-

3

Thanks I will try the patch tonight.

4

I have tried a few different kernels but they just boot to a black screen.

5

A black screen could be a good sign, meaning your GPU is disconnected now. SSH into the box from another machine and check your iommu and lspci again.

6

I don’t have a spare computer, but I could get in using nomodeset. The IOMMU groups have changed. What do I need to do to get it to boot correctly now?

7

check out this script, it will give a listing of what’s in each group, please post that here
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Ensuring_that_the_groups_are_valid

also post an updated lspci -nnk for the video cards (without any grep please)

In your bios setup, which card / slot do you have selected as the “boot” card? That might be causing your black screen issue?

8

Thanks,
I just got it booting to the desktop and now I am setting up the VM

9

I have Windows installing in the VM now.

I got my Ubuntu to boot to desktop, by booting into nomodeset and changing the drivers to nouveau rebooting back into nomodeset changing the drivers back to the NVIDIA drivers then it booted in to the desktop correctly.

1 Like