The state of PCI-E passthrough on linux in 2021

its been a long time since ive looked at graphics card passthrough things and im currently shopping around for a full system upgrade. im eying up ryzen 3000 chips and intels 10400f chip for my next CPU. i doubt i would have any problems picking the ryzen for this job, but i am aware that motherboards may pose difficulties with PCI groupings. how are the intels, and what should i be aware of when shopping for boards for either CPU? is it the case that i need to buy one of a small selection of motherboards with a specific chipset, or are things better in 2021 for this?

I’m no expert… but I can say the iommu grouping on my x570 crosshair are perfect for VFIO PCIe passthrought. No sweat at all. It’s the only one I tried it on, because it’s the board i had when i discovered the magnificent world of VMs. So I can’t really say if other board aren’t as good lol, just that that one rocks!

Just got it working a few weeks ago myself using a B450. Things are looking pretty good. The reset patches for AMD cards are pretty mature at this point as far as I understand, and Nvidia’s driver lock has work arounds.

As for motherboard selection, if your willing to apply the ACS override patch, I’m pretty sure there’s very few boards that WONT work at least to some extent. One thing I wish I went for was a board that supported bifurcation of the 16x slot. So my setup has the hypervisor GPU run off a x4 link through the chip set. However that is more of a nice to have than a need to have.

ive got a friend with experience with ESXi advising i go intel for higher stability with virtualisation do to intels more mature platforms. should i be aware of anything that may dissuade me beyond AMD just being faster these days, or any concerns specific to intel?

is the ACS override patch an AMD thing? i apologies for asking to be spoonfed, im trying to get back into the loop of things. is there even much benefit to having a board that cant bifurcate the CPU x16 slot if only one card is doing anything beyond rendering a desktop?

Intel tends to have less headaches, but amd is getting really close.

I’ve done passthrough on skylake, first gen ryzen, threadripper and 3rd gen. The change in maturity between first gen and 3rd gen is astonishing.

While it’s not plug and play, it’s damn close.

What it really boils down to now is just platform maturity.

thanks, it certainly doesnt help put me off intel that theyre currently heavily discounted and ryzen 3 is inflated either.

anything i should know about board choices with intel?

Well, I haven’t used intel since skylake, so I can’t say, frankly.

If you want to move on intel, there will be fanboys telling you not to, but frankly, just pick the right tool for the job. Wish I could be more help with the selection process though.

no worries, and thanks for sharing what you can. i still feel like im prodding in the dark atm but when ive got a clearer picture of what ill need ill pull the trigger on the hardware and give this a go.

1 Like

Let’s put it this way. It’s hard to pick hardware that is outright incompatible at the moment.

There are some edge cases, but for the most part, consumer grade hardware all supports passthrough at some level. You might have to perform some software black magic to get it to work just right, but it will almost always work properly given the right tweaks.

how are things in terms of USB devices with standard methods of virtualisation? ive now discounted ESXi after realising not only do i need to run 2 guests with it on account of how it works, but i cannot afford a motherboard with 3 x16 slots and enough useable x1 slots for all the expansion cards id need in order to not have the hypervisor throw a tantrum for being denied video output privileges and the privilege of having working keyboards on my guests.

off the top of my head the checklist i need is 1 GPU per OS, inc host. 1 sound card passed through to the windows guest, the host using onboard. do i need USB cards to pass through to the guest, or can i give the guest a keyboard, mouse, and maybe a front port for my controller and have that just be fine to game on?

No the ACS override patch is not an AMD thing only. However, do some research on it before hand, because there’s side effects to it, even if it does open up more IOMMU groups. For me, I’ve been passing my USB devices through virt manager, no additional PCI USB cards needed, however its a little awkward. If true seamless hot plugging is what your after, USB cards are the best way to accomplish it.

If you want to pass through usb ports, you have to pass through the controller. You’d be best off with a dedicated controller if you’re doing that.

You can do usb redirection, but I’m not sure how well that’ll work. I’ve never really used it for latency sensitive devices.

Side effects only show if you’ve got crossfire capable gpus or odd enterprise hardware that uses pci p2p features.

The acs allows devices to communicate p2p, and prevents them from being on different vms. That isn’t really a problem because the p2p communication isn’t really implemented often at all.

That said, we might be starting to see it with the likes of the ps5 which loads textures direct from the ssd to gpu memory. Not sure if they’re actually using pcie p2p comms for that or what, but it’s worth noting.

Ah, was fuzzy on the details for the ACS patch. Thanks for clearing that up.

I’ve been using USB host pass through through lib-virt. Great results. I’ve been using a VR headset, so defiantly a latency sensitive application and it’s been working great for me. Honestly can not tell the difference between running windows natively.

That’s good to hear. I think that I’ll be running my Index in a VM when I get my hands on a second card that I can properly watercool. I was worried about latency with USB passthrough.

Yeah how hard are these looking glass patches to apply to a 5700xt for vendor reset? Been thinking about it lately whether i should sell my 5700xt and get a 6700 or 6800 to run as 2nd gpu or subject myself to some possibly wasted time.

The vendor reset patches are not related to Looking Glass in any way, to my knowledge.

Vendor reset would be a kernel patch, which means you have to compile your own kernel.