Using ACS to passthrough devices without whole IOMMU group

Software & Operating Systems Linux
1

I have gotten my system finally running and configured for PCI passthrough. However, I am struggling to passthrough the proper amount of USB devices without passing an entire USB controller. Unfortunatley my IOMMU groups are as such:
https://pastebin.com/P7XGTmUA
I have applied the ACS patch, (using the linux-vfio kernel from AUR) but still virt-manager prevents me from creating a virtual machine with individual devices from a IOMMU group, reporting that

Please ensure all devices within the iommu_group are bound to their vfio bus driver.

Is there something I am doing incorrectly?
Relevant info:
Processor: AMD Ryzen 1800X
Motherboard: ASRock X370 Taichi
Host GPU: AMD Radeon Rx 580
Guest GPU: AMD Radeon Rx Vega 64
Host OS: Arch Linux
Guest OS: Windows 10
Kernel Version: 4.14.9-1-vfio (From AUR)
/etc/default/grub.cfg: https://pastebin.com/UxHXXDta
/etc/mkinitcpio.conf: https://pastebin.com/gGbG6x02
/etc/modprobe.d/vfio.conf: https://pastebin.com/NxPFSrFs
Grub checked to make sure options are being loaded at kernel runtime
Any other info I can offer I would be happy to provide. Thanks in advance everyone!

2

You can’t pass through individual devices within IOMMU groups.

i.e. If your network card and USB controller are in the same group, it’s a both or none situation.

You can usually passthrough separate USB devices in virt-manager by going to Add Hardware > USB Host Device Is this what you are trying to do?

If so and it’s failing, then I suspect it’s because your USB controllers are grouped with other hardware.

3

Seems I fundamnetally misunderstood the ACS patch then. Is there any workaround for this? If there isn’t, I am SOL. The majority of my gaming at this point makes use of greater than 4 USB devices. In the Arch wiki, it shows some additional ACS options, using

pcie_acs_override = id:nnnn:nnnn

Could I use this to further break out my IOMMU groups?

4

The ACS override patch tries to separate devices into their own IOMMU group. Doesn’t always work.

What USB controller are you trying to pass through? the ones at 12:00.3 and 03:00.0 cannot be passed through.

If I understand correctly, the one at 0c:00.0 is attached to a PCIe slot? If so, what slot? You may want to try switching it around. (if it shows up with the Vega 64, that will be fine as well)

You could try pcie_acs_override=downstream as well, but that gets into dangerous territory.

On the Taichi, have you updated the BIOS? I’ve got the Taichi as well, but with the 1700 instead. My IOMMU groups look significantly different. Later BIOS revisions have better IOMMU groupings.

5

I’l check again on the bios. I have actually now switched to pcie_acs_override=downstream,multifunction and I have all the iommu groups I could ever need :smiley: . Whats particularly dangerous about downstream? I had already had that enabled, as that’s how the arch wiki describes setting it up.

6

The ACS override patch doesn’t actually change the groupings. It just changes how they appear to Linux.

This means that on an electrical level, the devices are still grouped and can communicate with each other. This can play havoc on stability.

For example: If you have two AMD cards in the same electrical IOMMU group and you apply the ACS patch, when you load the guest GPU driver, both the host and guest will crash because the GPU looks in the IOMMU group, sees the other GPU (for the host) and tries to communicate with it. Because the host GPU is running a different driver (Linux), they can’t properly communicate and they both get corrupt and crash.

For the record, it’s not dangerous in a permenant fashion, just dangerous as far as stability goes. While I don’t know enough about PCIe standards, protocols and protections go, I don’t think this could cause you to fry a device.

I’m using my Taichi without the ACS override patch, but I’m not passing through USB. I’d give a bios update a try first. Don’t use ACS override if you can help it.

1 Like
7

@cheronobyl Here’s my IOMMU grouping with a USB controller in the top x1 slot. No ACS patch here.

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric Device 18h Function 6 [1022:1466]
IOMMU Group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 11 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
IOMMU Group 11 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
IOMMU Group 11 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
IOMMU Group 11 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
IOMMU Group 11 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 11 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 09:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 11 0b:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 11 0c:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
IOMMU Group 12 0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/580] [1002:67df] (rev e7)
IOMMU Group 12 0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580] [1002:aaf0]
IOMMU Group 13 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [10de:1b82] (rev a1)
IOMMU Group 13 0f:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 7 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 7 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 7 11:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 7 11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) USB 3.0 Host Controller [1022:145c]
IOMMU Group 8 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 8 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 8 12:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 8 12:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)

USB controller is at 0c:00.0

I’m going to try the bottom x16 (x4 electrical) slot a bit later and report my findings.

8

Seems to be about what I had experiences, no USB devices in individual groupings unless im missing something. Also, I found that the top and bottom PCI slots are in the same IOMMU grouping.

9

bummer

Theoretically, the ACS patch should solve this, but let me put a bit of work into this and see if I can get it to work before I start making lofty claims.

10

The ACS patch does at least split it from group 11, but I am unsure if it makes them into individual groups. I don’t have another PCI card to test with at the moment, so I cannot see for sure if it groups itself separate. At this point, I have everything working flawlessly and was even able to play some Robo Recall on my Rift from withing the Guest system (part of why I need so many USB devices and why they needed to be passed through).

11

This got my pfSense setup working in Proxmox, thank you!

I did notice a single page fault but it seems to have been a total fluke

12

I was also having trouble with not having enough IOMMU groups. I needed to passthrough a PCI USB 3.0 card for my VM.

adding pcie_acs_override=downstream,multifunction + enabling ACS from BIOS menu gave me enough IOMMU groups to satisfy my req. Fedora 36, Ryzen 7 2700 and Gigabyte Aorus B450 Wifi.

13

Please keep in mind that this doesn’t actually change the IOMMU groups, it only allows you to jump the fence, so to speak.

The IOMMU groups are still there, and P2P communication could happen, causing corruption and issues, but this is a rarely-used feature of the PCIe fabric.

I used to use it a lot back when I was running passthrough, and it always worked fine, it’s just one of those things that I believe should come with a bold-font asterisk.

1 Like