Ryzen IOMMU: PCIe Passthrough works, BUT... | Level One Techs

I am happy to report that the ACS patches still work with Kernel 4.10+ (Kernel 4.10+ is recommended for Ryzen due to additional functionality of the Ryzen hardware). However, it is a bit weird.

On both the ASRock Taichi X370 (Review out at youtube.com/level1techs) and the Aorus Gaming 5 from Gigabyte, the two CPU-connected PCIe x8 slots are both in IOMMU group 2.



This is a companion discussion topic for the original entry at https://level1techs.com/video/ryzen-iommu-pcie-passthrough-works
18 Likes

Thank you on behalf of all who use PCI Passthrough.

2 Likes

Thanks for the info. I myself do not feel up to trying that out yet, haven't found the motivation to torture myself with broken packages and obscure settings yet lol. I do hope the bugs get ironed out quickly, especially UEFI bugs, I'm having some pretty annoying ones on my Ryzens right now.

Is it something that can eventually be fixed with CPU microcode, UEFI, or Kernel update? Or is it a permanent problem that won't get fixed until Ryzen 2?

should just be kernel updates

but will have to wait a bit/potentially quite a bit for those kernel changes to trickle down to stable distros.

like.. just installing linux on ryzen currently is an issue depending on distro cause most have kernels <4.8~4.9 for the install media even for new releases like fedoras, 4.10 is only from jan so probs wont hit for like 6-12months for alot of dists

best freeze frame ever...

... sorry @wendell

12 Likes

I just read up on this whole topic a bit, and what I suspect is happening (especially with the combination of amd cards) is the following:

Using the acs override you basically just tell the kernel: "yeah, it's all isolated, don't listen to what the hardware says"
For simple situations this would actually work, the bigger problem comes when your cards try to DMA to each-other.

Amd for a while now has supported "bridgeless crossfire" (i believe it's called crossfireX?) on their gpus.
this bridgeless mode relies on pci-e DMA between the cards for communication.
What i suspect is happening is that when you install the native amd drivers on windows, somehow the 2 cards detect each-other and try to establish some basic knowledge of each-other expecting to be controlled by the same driver, this confuses both drivers/gpu's causing a complete crash. (maybe related to the state of crossfire support in the linux driver?)

so basically i suspect that we are fscked for the "similar cards" scenario unless real/half decent acs can be "patched in" trough some sort of microcode/chipset firmware/bios upgrade. (or maybe — if my hypothesis is correct — if we can find some sort of "no-crossfire" flag in the windows drivers)

for the different gpu setup: not sure what's going on there.
I suggest trying to replicate the "hangs after a while" issue with linux in the vm (pick ubuntu or another distro that doesn't give you headaches with proprietary nvidia drivers)
If it replicates under linux, there is a chance that we can get some debug information from the vm's kernel to figure out what's happening. If it doesn't then it might be some weird windows issue :-(

Good luck hacking on all this stuff, if you don't get trough it, in a month or so i'll have my own ryzen system to throw some tests at.

12 Likes

I am sure this is the issue, inter-card communication.

I have asked the motherboard vendors to give us an "initailize gpu first" option to pick the gpu attached to the x370. Apparently B350 also has an x4 slot tied to the nvme lanes so that might be a really good choice, too, if the graphics card there can be init first.

2 Likes

Really interesting. I am all but sure I'm getting a 1700(X) still deciding / saving. It is good to see Gigabyte is working with Wendell and I hope they can offer some support to linux users from their side.

I wonder how the APU's will tackle this on Ryzen R5 and R3. APU and passthrough GPU to a VM. Interesting times.

I have the same problem with lockups on my B350 board with a RX480 and a Radeon 7850. If I have those two cards in my system, my kernel won't boot up fully and will have a kernel panic. These aren't even in the same IOMMU group, one is on a group 0 port hooked to the B350, the other is direct to the CPU in group 2.

I get a flood of "AMD-Vi: Completion-Wait loop timed out" followed by a "Kernel panic - not syncing: Fatal exception in interrupt"

If I instead use my Radeon 5870 with my RX480, the system will boot up just fine. One thing, the R9 Fury, 7850, and RX480 are all GCN cards, while the 5870's a VLIW5 card instead.

I think I'm going to skip the IOMMU Passthrough for now. I could probably get it to work on the B350, but I don't want to lose linux graphics performance for it by putting my linux gpu on the slow lanes. I have some other plans in mind.

Yeah, you almost always want different vendors (Read: one nv and one AMD card) on ACS patched systems. I could have told you that.

It's not just a problem with an ACS patched system. I'm not ACS patched and still get kernel panics with both of those cards installed.

Hey. I have a b350 motherboard w/ R7 1700. Here are the iommu grouping from the machine with only 1 graphics card installed:

Seems to me that if I install a second card into the x4 slot, the machine should keep the two separate as one is running on the 3.0x16 from the mobo, and the other is running in the 2.0x4 from the chipset.

I don't have a second card handy that can fit in the x4 slot, but will update you guys on how it goes once i do

You can install anything in the x4 just to see what it does.

Problem will be no way to boot from the x4 first and leave the x16 uninitialized.

And the x,4 will be put into group 0 I'd guess

2 Likes

What did Gigabyte say in your chat to them Wendell. Can they do anything on the motherboard level or is this an architecture is not made to do this thing ?

They are looking into it. I think all those bridges there is a good sign, and it is a software thing probably.

2 Likes

I do hope Gigabyte and others are watching us lowly tech people and helping us out. It would be a make friends forever moment in history like Voodoo GFX cards blew my mind on quake and I will never forget.

1 Like

So far ryZEN seems rather lackbuster
AMD was known for years as "cheap" "almost as good as Intel" "you could OC a lot"
but it had issues using more power then Intel and more heat ....

Now we have buggy firstgen mobos...
Prices arnt that cheap either in Australia
( I am going to guess same is in JP/NZ/Philippines also)

  • AMD Ryzen 7 1700 8-Core Socket AM4 3.0GHz CPU
    Processor with Wraith Spire Cooler - AUD$469

  • MSI B350M Gaming Pro AM4 mATX Motherboard - AUD$129

  • Kingston 8GB (2 x 4GB) KINHX426C15FBK2/8 2666MHz
    DDR4 Non-ECC CL15 DIMM HyperX FURY Black - AUD$97

So that is AUD$695 just for the CPU/MOBO/RAM before anything else
Then if you wanted a "budget build" :-

  • Western Digital blue 1TB SATA3 HDD 64M Caviar blue wd10EZEX - AUD$65
  • Gigabyte Radeon RX 460 Windforce OC 2GB - AUD$$145
  • Silverstone ST50F-ES230 Essential 230V 500W psu 80+ - AUD$55
  • Silverstone PS08B Micro ATX Case - AUD$45.0
  • 20 bucks DVDRW if you want one......
  • 20 bucks generic kbm

Which ends up as AUD$1045

ohh and that isnt shipping (AUD$15-25) and
system build cost (normally AUD$50-55 bucks by a shop)
or windows that is like AUD$135-AUD$200

So it could end up AUD$1300

I cant shake the feeling youd be better off getting an
G4560 or kaby lake i3 and ocing them with an skylake mobo
and firmware flashing the skylake z170 mobo to support
the kaby lake cpus , which would save like 200 - 300 bucks

Depends on use case, and the r7's aren't budget parts.

While a lack of proper IOMMU isolation isn't ideal, the ACS patch requirement is pretty common on consumer chips. (pretty much all skylake stuff needs it for anything extensive.)

I think it's a great option if you're doing real work. If you just want to game, sure, go intel all the way for now, but Ryzen definitely has valid use cases outside of the hobbyist stuff like gaming and overclocking.