USB 3.0 controller PCIe passthrough

Software & Operating Systems VFIO
1

So I am trying to passthrough a USB 3.0 add in card to my VM in order to be able to switch keyboard and mouse between host and guest by using a small KVM switch. But somehow it doesn’t work as expected.

I might not know where to start and which information you need in order to help me since I am new to the Linux universe but I’ll try to do my best.

I went with Fedora30 WS (Kernel version 5.0.9.301 with ACS patch) and try to use it as my daily driver in the future. But I don’t want to go completely without Windows for now. (Partly because of my gaming passion) So I plan to use VFIO to passthrough a few PCIe devices to my VM in order to enjoy near native performance without having to reboot all the time.
I already managed to passthrough an NVMe SSD, a SATA SSD (block device) and my GTX 980Ti (although its fans always spin up to a 100% as soon as I shut down the VM) to the VM.

But the USB 3.0 add in card is something I can’t handle.
I tried two different ones (inateck KT2001 and StarTech PEXUSB3S24) and they cause different problems.
Lets start with what both of them have in common:

  1. Massively slow down startup and shutdown of the VM (without passing them through the VM starts up lightning fast)

  2. They don’t work inside the VM (and I am sure they work on the host OS)

  3. Both got power because the LEDs of attached keyboards or flash drives light up

The StarTech card doesn’t even show up in the Device Manager inside the VM.

The inateck with the ASMedia chipset does show up in the Device Manager and works properly according to Windows even though it doesn’t work at all inside the VM.
After shutting down the VM the inateck card locks up my whole libvirt process. No matter what I try, I can’t kill the process in this case and I can’t even gracefully shutdown the host OS. A hard reset is required to revive libvirt.

With that said I think I just dump some information about my configuration in here.

Hardware:
CPU: AMD Ryzen R5 2600
MB: ASUS ROG Strix X470-F Gaming (BIOS Version 4207)
RAM: 32GB GSkill Something @ 3200MHz CL16
GPU: AMD Radeon RX 580 Red Devil 8GB by PowerColor
GPU2: Nvidia GTX 980 Ti Gaming G1 6GB by Gigabyte
USB add in card: inateck KT2001 or StarTech PEXUSB3S24

Note: I had to downgrade the BIOS from 5204 because it broke VFIO passthrough for me by introducing some weird power state D3 problem with my GTX 980Ti which resulted in “Unknown PCI header type ‘127’”
But thats fixed with the older 4207 BIOS.

Other than that I think my IOMMU Grouping is fine: (The ACS patch helped a lot with that)

IOMMU%20Groups
IOMMU Groups.png1314×506 37.6 KB

Also the vfio-pci driver manages to capture all the devices I want to passthrough. (Even though some people say this is only necessary for the GPU; other device would also work with other drivers)

VFIO-PCI
VFIO-PCI.png1236×561 32.9 KB

And last but not least I think my KVM domain looks also fine
win10-domain.txt (6.2 KB)

I am out of ideas and would greatly appreciate any attempt to help me with my USB add in card passthrough. (Or even with my GPU fan problem I mentioned on the side.)

Thanks in advance.

2

@FurryJackman had issues with usb cards, but found one that worked, see his thread here-

Jack's Hardware: The Ultimate VFIO USB 3.0 Controller

3

@ TheCakeIsNaOH Thanks for your answer.

I also stumbled across this thread during my research. He is recommending the exact same StarTech card I am using.
Still no luck with that unfortunately.

4

Sorry, I didn’t look close enough at the models.

It is posible that ACS is not working correctly on your motherboard, so the patch makes it unstable, but other then that, I don’t know.

5

X470 chipset. That’s your problem because everything is under a single IOMMU group on that chipset, and you’re probably using the x1 slot going to the chipset.

Use direct lanes going straight to the CPU. Use a M.2 to PCI-E x4 riser and then put your Startech on that so that the lanes go direct to the CPU.

Either that or upgrade to a X570 chipset mobo because the IOMMU separation is better on that set of mobos. Asrock X570 Taichi gets my early vote.

1 Like
6

@FurryJackman Your hint seems to be the solution to my problem.

I just removed my GTX card from the x16 slot and put my StarTech USB card in there instead.
And suddenly the USB Controller works fine inside the Windows VM.

The X470-F Strix Bord has two M.2 slots. One of them seems to be connected directly to the CPU and the other one pulls its lanes from the chipset.
So your suggestion using an M.2 to PCIe adapter might work if I use the M.2 slot which uses the CPU lanes.

BTW: Passing through my NVMe SSD works with both slots.

My next step is to obtain such an adapter. I’ll keep you updated on that.

Sidenote: In order to get the fan speed of my GTX GPU under control while the VM is turned off I plan to install the proprietary Nvidia drivers.
For switching between the vfio-pci and the nvidia drivers I might use a libvirt hook script which listens to the QEMU state.
Yesterday I tried to do it with nouveau instead of the proprietary drivers but I couldn’t get fan control working with this driver.

But that is probably a topic for another thread.

2 Likes
7

Nouveau is a hopeless cause. If fan speeds work with the proprietary drivers, that’s the solution to stick to.

If it were a 780 Ti, it’s a different story, but because you’re running a 980 Ti, that doesn’t work at 3D clocks AT ALL using Nouveau. Nouveau would have a HUGE performance penalty for being locked at 2D clocks, and they need signed Nvidia firmware to reclock, which is very unlikely to ever happen. The documentation may be out, but it still needs signed firmware.

1 Like
8

One solution would be to pass the CPU integrated USB. Had that working on 1920X so it should be possible for you too.

1 Like
9

This is what I’ve done for running my Windows 10 VM on x470 as well. There’s an IOMMU group that has USB ports, a security device, and sometimes some other bits

1 Like
10

I am sure that the devices were split in later AGESA for 1920X and 1700X.
Maybe the new 1.0.0.3 reverted it. 3700X has them back in the same group.

11

Ah ok. My 3700x shuffled around some of the groupings as well. On your board, have you tried putting the USB card in the middle x16 slot? The 2nd one with the metal border around it? That should be connected to the CPU and allow you to passthrough the device. It’d force you to put the host GPU on the bottom slot, which would stink a bit, but would fix the passthrough issue

12

Wendell said the ABB AGESA fixes more issues with IOMMU, but it’s only rolled out with some BIOS partners and not all of them.

13

ABB did not split the USB, SATA, Audio and PSPCPP on 3700X.
I dont have older CPUs to check if they are affected as well.
Fortunately the X570 chipset allows passing its integrated USB controllers.

14

Damn, I thought the IOMMU groupings had gotten better with the newer chipsets. In my case one of the USB controllers was in a separate group so I was able to pass that through and use a HUB.

I did have the same issue with my 390 and the fans being on until I started the VM. I do recall that once the VM was shut off the fans did stay off. I never resolved it and eventually ended up swapping the cooler out with a kraken G12 and AIO. Still curious on a proper fix for it though.

15

The CPU secondary IOMMU groups regressed upon release of Ryzen 3000 on Ryzen 3000 CPUs, but the primary IOMMU groups are fine. The best setup right now is actually a 2700X and a X570 mobo. IOMMU groups on the CPU for secondary things I assume were not affected on Ryzen 2000 on X570.

16

I got that working for me as well now.
And I also received my M.2 adapter yesterday.
Both options work flawlessly.

Thank you all. You are awesome!

1 Like