VFIO - USB Controllers in separate Groups with same ID

Software & Operating Systems Linux
, ,
1

Hello everyone,

I am trying to passthrough 1 of my onboard ASMedia USB 3.1 Controllers to a VM, while they both are in separate groups they seem to have the SAME id. How is this even possible?

IOMMU Group 14 03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU Group 15 04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]

and here is the full iommu grouping if needed

IOMMU Group 0 00:00.0 Host bridge [0600]: Intel Corporation Sky Lake-E DMI3 Registers [8086:2020] (rev 04)
IOMMU Group 10 00:1d.2 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #11 [8086:a29a] (rev f0)
IOMMU Group 11 00:1f.0 ISA bridge [0601]: Intel Corporation X299 Chipset LPC/eSPI Controller [8086:a2d2]
IOMMU Group 11 00:1f.2 Memory controller [0580]: Intel Corporation 200 Series/Z370 Chipset Family Power Management Controller [8086:a2a1]
IOMMU Group 11 00:1f.3 Audio device [0403]: Intel Corporation 200 Series PCH HD Audio [8086:a2f0]
IOMMU Group 11 00:1f.4 SMBus [0c05]: Intel Corporation 200 Series/Z370 Chipset Family SMBus Controller [8086:a2a3]
IOMMU Group 12 00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (2) I219-LM [8086:15b7]
IOMMU Group 13 02:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 14 03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU Group 15 04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller [1b21:2142]
IOMMU Group 16 16:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
IOMMU Group 17 16:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
IOMMU Group 17 16:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU Group 17 16:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
IOMMU Group 18 16:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:08.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:08.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:08.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:08.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:08.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 18 16:08.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 19 16:09.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 19 16:09.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208d] (rev 04)
IOMMU Group 1 00:04.0 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.1 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.2 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.3 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.4 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.5 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.6 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 1 00:04.7 System peripheral [0880]: Intel Corporation Sky Lake-E CBDMA Registers [8086:2021] (rev 04)
IOMMU Group 20 16:0e.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 20 16:0e.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 20 16:0e.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 20 16:0e.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 20 16:0e.4 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 20 16:0e.5 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 20 16:0e.6 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 20 16:0e.7 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 21 16:0f.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 21 16:0f.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:208e] (rev 04)
IOMMU Group 22 16:1d.0 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2054] (rev 04)
IOMMU Group 22 16:1d.1 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2055] (rev 04)
IOMMU Group 22 16:1d.2 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2056] (rev 04)
IOMMU Group 22 16:1d.3 System peripheral [0880]: Intel Corporation Sky Lake-E CHA Registers [8086:2057] (rev 04)
IOMMU Group 23 16:1e.0 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2080] (rev 04)
IOMMU Group 23 16:1e.1 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2081] (rev 04)
IOMMU Group 23 16:1e.2 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2082] (rev 04)
IOMMU Group 23 16:1e.3 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2083] (rev 04)
IOMMU Group 23 16:1e.4 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2084] (rev 04)
IOMMU Group 23 16:1e.5 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2085] (rev 04)
IOMMU Group 23 16:1e.6 System peripheral [0880]: Intel Corporation Sky Lake-E PCU Registers [8086:2086] (rev 04)
IOMMU Group 24 17:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
IOMMU Group 25 18:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
IOMMU Group 26 18:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
IOMMU Group 27 19:00.0 VGA compatible controller [0300]: NVIDIA Corporation G73 [GeForce 7300 GT] [10de:0393] (rev a1)
IOMMU Group 28 1a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
IOMMU Group 28 1a:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
IOMMU Group 29 64:00.0 PCI bridge [0604]: Intel Corporation Sky Lake-E PCI Express Root Port A [8086:2030] (rev 04)
IOMMU Group 2 00:05.0 System peripheral [0880]: Intel Corporation Sky Lake-E MM/Vt-d Configuration Registers [8086:2024] (rev 04)
IOMMU Group 2 00:05.2 System peripheral [0880]: Intel Corporation Device [8086:2025] (rev 04)
IOMMU Group 2 00:05.4 PIC [0800]: Intel Corporation Device [8086:2026] (rev 04)
IOMMU Group 30 64:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
IOMMU Group 30 64:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU Group 30 64:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
IOMMU Group 31 64:08.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
IOMMU Group 32 64:09.0 System peripheral [0880]: Intel Corporation Device [8086:2066] (rev 04)
IOMMU Group 33 64:0a.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
IOMMU Group 33 64:0a.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
IOMMU Group 33 64:0a.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
IOMMU Group 33 64:0a.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
IOMMU Group 33 64:0a.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
IOMMU Group 33 64:0a.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
IOMMU Group 33 64:0a.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
IOMMU Group 33 64:0a.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
IOMMU Group 34 64:0b.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
IOMMU Group 34 64:0b.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
IOMMU Group 34 64:0b.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
IOMMU Group 34 64:0b.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
IOMMU Group 35 64:0c.0 System peripheral [0880]: Intel Corporation Device [8086:2040] (rev 04)
IOMMU Group 35 64:0c.1 System peripheral [0880]: Intel Corporation Device [8086:2041] (rev 04)
IOMMU Group 35 64:0c.2 System peripheral [0880]: Intel Corporation Device [8086:2042] (rev 04)
IOMMU Group 35 64:0c.3 System peripheral [0880]: Intel Corporation Device [8086:2043] (rev 04)
IOMMU Group 35 64:0c.4 System peripheral [0880]: Intel Corporation Device [8086:2044] (rev 04)
IOMMU Group 35 64:0c.5 System peripheral [0880]: Intel Corporation Device [8086:2045] (rev 04)
IOMMU Group 35 64:0c.6 System peripheral [0880]: Intel Corporation Device [8086:2046] (rev 04)
IOMMU Group 35 64:0c.7 System peripheral [0880]: Intel Corporation Device [8086:2047] (rev 04)
IOMMU Group 36 64:0d.0 System peripheral [0880]: Intel Corporation Device [8086:2048] (rev 04)
IOMMU Group 36 64:0d.1 System peripheral [0880]: Intel Corporation Device [8086:2049] (rev 04)
IOMMU Group 36 64:0d.2 System peripheral [0880]: Intel Corporation Device [8086:204a] (rev 04)
IOMMU Group 36 64:0d.3 System peripheral [0880]: Intel Corporation Device [8086:204b] (rev 04)
IOMMU Group 37 65:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
IOMMU Group 38 66:08.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
IOMMU Group 39 66:10.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch [10b5:8747] (rev ca)
IOMMU Group 3 00:08.0 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2014] (rev 04)
IOMMU Group 3 00:08.1 Performance counters [1101]: Intel Corporation Sky Lake-E Ubox Registers [8086:2015] (rev 04)
IOMMU Group 3 00:08.2 System peripheral [0880]: Intel Corporation Sky Lake-E Ubox Registers [8086:2016] (rev 04)
IOMMU Group 40 67:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
IOMMU Group 40 67:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 41 b2:05.0 System peripheral [0880]: Intel Corporation Device [8086:2034] (rev 04)
IOMMU Group 41 b2:05.2 System peripheral [0880]: Intel Corporation Sky Lake-E RAS Configuration Registers [8086:2035] (rev 04)
IOMMU Group 41 b2:05.4 PIC [0800]: Intel Corporation Device [8086:2036] (rev 04)
IOMMU Group 42 b2:12.0 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204c] (rev 04)
IOMMU Group 42 b2:12.1 Performance counters [1101]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204d] (rev 04)
IOMMU Group 42 b2:12.2 System peripheral [0880]: Intel Corporation Sky Lake-E M3KTI Registers [8086:204e] (rev 04)
IOMMU Group 43 b2:15.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU Group 44 b2:16.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU Group 44 b2:16.4 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU Group 45 b2:17.0 System peripheral [0880]: Intel Corporation Sky Lake-E M2PCI Registers [8086:2018] (rev 04)
IOMMU Group 4 00:14.0 USB controller [0c03]: Intel Corporation 200 Series/Z370 Chipset Family USB 3.0 xHCI Controller [8086:a2af]
IOMMU Group 4 00:14.2 Signal processing controller [1180]: Intel Corporation 200 Series PCH Thermal Subsystem [8086:a2b1]
IOMMU Group 5 00:16.0 Communication controller [0780]: Intel Corporation 200 Series PCH CSME HECI #1 [8086:a2ba]
IOMMU Group 6 00:17.0 SATA controller [0106]: Intel Corporation 200 Series PCH SATA controller [AHCI mode] [8086:a282]
IOMMU Group 7 00:1c.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #1 [8086:a290] (rev f0)
IOMMU Group 8 00:1c.2 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #3 [8086:a292] (rev f0)
IOMMU Group 9 00:1d.0 PCI bridge [0604]: Intel Corporation 200 Series PCH PCI Express Root Port #9 [8086:a298] (rev f0)

Anyone know how to fix this ID problem on the USB 3.1 Controllers?

Cheers!

2

You cannot “fix” them having the same ID. The ID is baked into the hardware. The address, however, will change depending on the motherboard.

The actual passthrough (libvirt or QEMU) configuration uses the address so it is not affected by having two of the same device. You will, however, have to use a script to use vfio for only one of the controllers instead of the normal /etc/modules entry.
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Using_identical_guest_and_host_GPUs

1 Like
3

Can you go into more details, the link seems to state that you can’t use scripts like this anymore in recent kernels.

Script installation

Note: This will not work as of Linux 4.18.16, since vfio-pci is compiled-in. Has to be done through an initcpio hook.

What is an initcpio hook?

Is it possible to use the old PCI-Stub method only for controllers like this since it didn’t use device id’s and instead used PCI Address?

4

Initcpio is what Arch uses for an initramfs. Since vfio is compiled in, the script needs to be run earlier in the boot I think, so the install method is different.

But I think you can use after boot unbinding+rebinding since it is not a VGA device. So just unbind one of the cards(echo 0000:03:00.0 > /sys/bus/pci/devices/0000:03:00.0/driver/unbind for example)
Then run the script, although I think you can distill it down to
echo "vfio-pci" > /sys/bus/pci/devices/0000:03:00.0/driver_override or similar. I think driver_override works after boot, but I am not sure.

5

Now that I am sitting here thinking about this, being that both of these controllers are going to go to separate VM’s and NOT leaving one on the host… I should be able to just specify 1b21:2142 in vfio config, it will bind both controllers to vfio-pci driver. Since QEMU uses PCI Address to bind too VM I should not have a problem with 1 controller going to VM 1 and the other to VM 2.

6

You generally shouldn’t need to do any setup for vfio to work on the usb controllers it should already work and bind to the vfio-pci driver when you attach it and launch a VM.

This is my controller on the host normally

michael@fedora ~ $ lspci -nnk -d 1022:43b9
03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller [1022:43b9] (rev 02)
	Subsystem: ASMedia Technology Inc. Device [1b21:1142]
	Kernel driver in use: xhci_hcd

and if I start my vm it binds to the vfio-pci driver

michael@fedora ~ $ lspci -nnk -d 1022:43b9
03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller [1022:43b9] (rev 02)
	Subsystem: ASMedia Technology Inc. Device [1b21:1142]
	Kernel driver in use: vfio-pci
1 Like
8

I can confirm this indeed works, I thought everything needed to be added to vfio config file… awesome it works and makes binding usb controllers easy!

1 Like