Question about IOMMU grouping

Hardware GPU
, ,
1

I just updated my Threadripper build from a Gigabyte Aurous TRX40 Extreme mobo to a MSI Creator TRX40 board, and I’m having issues getting the VFIO passthrough GPU, a Merc 6800XT. I’m wondering if the IOMMU grouping might be causing the issue.

The IOMMU groupings are as followed:

IOMMU Group 59 4c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 60 4c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 61 4c:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 62 4c:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]

Each device is in it’s own group, but will that work? I thought they all had to be in the same group?

Thanks in advance.

2

Now that is correct. If the VM is not working with the 4c:00.0 device (GPU) passed through, you can also pass through the 4c:00.1 (GPU Audio) device to the VM. The GPU devices are separeted on my machine as well. No problemo!

What kind of issue are you facing, you did not describe it at all?

1 Like
3

Thanks for the reply.

Basically I’m passing all four devices on the GPU and trying to use the GPU, and USB device as the peripherals input to the VM. The VM will start, but no video out or lighting on the KB and Mouse are visible.

4

Have you loaded the vfio-pci driver for all of those devices or is another kernel driver loaded of any of them?

5

I checked and here is the lspci -nnv output:

4c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1) (prog-if 00 [VGA controller])
Subsystem: XFX Limited XFX Speedster MERC 319 AMD Radeon RX 6800 XT Black [1eae:6701]
Flags: fast devsel, IRQ 11, IOMMU group 59
Memory at d0000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at e0000000 (64-bit, prefetchable) [disabled] [size=2M]
I/O ports at b000 [disabled] [size=256]
Memory at e1b00000 (32-bit, non-prefetchable) [disabled] [size=1M]
Expansion ROM at e1c00000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [200] Physical Resizable BAR
Capabilities: [240] Power Budgeting <?>
Capabilities: [270] Secondary PCI Express
Capabilities: [2a0] Access Control Services
Capabilities: [2d0] Process Address Space ID (PASID)
Capabilities: [320] Latency Tolerance Reporting
Capabilities: [410] Physical Layer 16.0 GT/s <?>
Capabilities: [440] Lane Margining at the Receiver <?>
Kernel driver in use: vfio-pci
Kernel modules: amdgpu

4c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
Flags: fast devsel, IRQ 5, IOMMU group 60
Memory at e1c24000 (32-bit, non-prefetchable) [disabled] [size=16K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

4c:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6] (prog-if 30 [XHCI])
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
Flags: bus master, fast devsel, latency 0, IRQ 43, IOMMU group 61
Memory at e1a00000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/8 Maskable- 64bit+
Capabilities: [c0] MSI-X: Enable+ Count=8 Masked-
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Kernel driver in use: xhci_hcd

4c:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:0408]
Flags: bus master, fast devsel, latency 0, IRQ 122, IOMMU group 62
Memory at e1c20000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [64] Express Endpoint, MSI 00
Capabilities: [a0] MSI: Enable+ Count=1/2 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [2a0] Access Control Services
Kernel driver in use: i2c-designware-pci

6

Those could be a problem. It depends how well, or even if, those can handle an automatic unloading. I guess you have specified the devices to load vfio-pci somewhere, can you add the IDs of the last two devices there as well?

7

Yes, I specified their ID’s in these two files:

/etc/modprobe.d/vfio.conf

options vfio-pci ids=1987:5018,8086:1539,1002:73bf,1002:ab28,1002:73a6,1002:73a4

/usr/sbin/vfio-pci-override.sh
#!/bin/sh
PREREQS=""
DEVS=“0000:01:00.0 0000:46:00.0 0000:4c:00.0 0000:4c:00.1 0000:4c:00.2 0000:4c:00.3”

for DEV in $DEVS; do
echo “vfio-pci” > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

8

Ok, did some testing and it seems that it is definitely the fact that the USB and serial device aren’t using VFIO drives. Now I wonder why?

9

how are you launching kvm/can you post the vm config?

10

I’m launching it via VirtManager.

I’m using the i440FX Chipset and
UEFI firmeware type: UEFI x86_64: /usr/share/edk2/ovmf/OVMF_CODE.fd

I also tested rebinding the devices to VFIO.
#!/bin/sh
PCI_HID=“0000:4c:00.2”
echo -n “$PCI_HID” > /sys/bus/pci/drivers/xhci_hcd/unbind
echo -n “$PCI_HID” > /sys/bus/pci/drivers/vfio-pci/bind
PCI_HID=“0000:4c:00.3”
echo -n “$PCI_HID” > /sys/bus/pci/drivers/i2c-designware-pci/unbind
echo -n “$PCI_HID” > /sys/bus/pci/drivers/vfio-pci/bind

Still no go…

11

Can you do a ‘virsh dumpxml’ ?

12

Here’s the XML output:

<domain type="kvm">
  <name>win10</name>
  <uuid>fa628b8f-4199-4147-a438-b1dc9eaf9b15</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">8388608</memory>
  <currentMemory unit="KiB">8388608</currentMemory>
  <vcpu placement="static">8</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-5.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
    <bootmenu enable="yes"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
  </features>
  <cpu mode="host-model" check="partial">
    <topology sockets="1" dies="1" cores="8" threads="1"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/win10-2.qcow2"/>
      <target dev="sda" bus="sata"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </controller>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </sound>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x46" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4c" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0a" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4c" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0b" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4c" slot="0x00" function="0x2"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0c" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x4c" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0d" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </memballoon>
  </devices>
</domain>
13

So I did some more tweaking and I still get a blank screen when I start the VM and the keyboard and mouse still don’t indicate that they get power.

I’m using the USB c port on the 6800XT connected to the L1tech DP1.2 KVM to a hub thats worked before. I don’t know what to do next as all indications are that VFIO drivers are being loaded.

14

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.