VFIO AMD ThreadRipper 2019 Notes ( Near Bare Metal Performance )

[saysay]

These are my notes to get AMD ThreadRipper up and running after VFIO setup. I have added Arch PKGBUILD files to build on the latest kernel. I have also added a start script I use.

The biggest tip is to use “Virtual network sound card for Microsoft Windows” that uses ivshmem. Very low latency. Maybe we could add this code to looking-glass.

VFIO AMD ThreadRipper 2019 Notes ( Near Bare Metal Performance )

[SgtAwesomesauce]

Thanks for sharing your notes! Looks great! I should post mine to compare.

Maybe we could add this code to looking-glass.

Out of scope.

The biggest tip is to use “Virtual network sound card for Microsoft Windows” that uses ivshmem. Very low latency.

Definitely gonna check that out though.

[gordonthree]

So tempting to try again but my x399 motherboard is a dud. Most of my slots don’t support IOMMU for some reason.

Maybe next year’s TR update from AMd will bring new boards to the field and I can upgrade.

[SgtAwesomesauce]

What board do you have?

Most of my slots don’t support IOMMU for some reason.

How is that possible?

[gordonthree]

Giga Designare x399 … The first two slots off the cpu, PCIEX16_1 and PCIEX8_1 just don’t show up in any iommu groups, and their pci bus addresses are “weird” compared to cards in the rest of the slots.

I posted about it a while back, but gave up after getting frustrated and just wanting the rig working. I was able to pass-through a card in PCIEX16_2 but using that slot is not ideal for the layout I want.

post here

X399 pass through more than one GPU?

System is a new TR2 build; Designare Ex x399, 64gb Corsair 4x16, Ryzen TR2 2950x, bios F11 GPUs x16 top slot GeForce GT310 for host, x8 slot under it AMD Firepro 59xx, bottom x16 slot Asus 1080 Strix OS; multiple, tried Fedora, Debian, esxi 6.7, unraid 666 Only GPU I can pass through is the one in the bottom x16 slot. The other cards don’t even show up in iommu groups. It’s not specific to the 1080, any of the cards in that slot are able to pass through. I get a vfio-pci error -22 for the card in the top x8 slot if I try to force it to bind to that driver. No idea what error -22 means.

[SgtAwesomesauce]

So glad I didn’t buy that board then.

[section279]

Umm I don’t think you’re doing something correctly as this thread on unraid shows slots 1 and 4 work: Gigabyte X399 Designare EX Threadripper

[gordonthree]

Just bad silicon I think for me.

The pcie addresses are different in that post. All of mine start with a 42:0:0 instead of 0:0:0 … I found a Linux kernel dev mailing list that explained what is happening but it’s over my head.

[lesser]

Any Benchmarks ran inside the Windows VM?

[gordonthree]

feeling masochistic, I booted the latest unraid and wiped the drives on my ripper. GTX 1080 in slot 1, acs override on, vfio-pci.ids excluded … try to pass through, vm hangs at boot. same as it ever was unfortunately.

[section279]

Any chance of walking through and diagnosing the problem with you?

1. What BIOS are you on?
2. Have you enabled SVM, ACS, and IOMMU in the BIOS?
3. What kernel revision are you using? Has it been patched for ACS override?
4. What boot/kernel flags are you using? (you need at least amd_iommu=on iommu=pt)
5. What is the output of dmesg | grep -i -e DMAR -e IOMMU and dmesg | grep AMD-Vi?
6. What is the output of lspci -tv?
7. What is the out out of lspci -vv for your GPUs?
8. What does find /sys/kernel/iommu_groups/ -type l output?
9. Can you provide your guest XML file?

Logs from dmesg when you run your vm are also appreciated.

Feel free to start your own thread with this info, and I’m happy to run through troubleshooting with you. I would however recommend trying the latest Pop!_OS guide posted by Wendell, as that is fairly helpful.

[anthr76]

Thanks for this ! 2950x user here on MEG Creation.

I mostly got a near ideal setup, but there was some pointers in here that were very helpful. I’m using Arch’s Linux 5.2 atm and it’s not playing nice. I’m going to give your kernel a try

[gordonthree]

Thanks for the offer. I gave up pretty quick on this. I copied my config from previous attempt with unraid.

[nothingburger3001]

The first two slots off the cpu, PCIEX16_1 and PCIEX8_1 just don’t show up in any iommu groups

Hmm, they do for me. F12e bios: https://pastebin.com/raw/kjc7BARn
1st X16 slot is group 11, 2nd X16 slot is group 2.

[gordonthree]

Here’s the debug info from months ago when I tried this under Fedora

Vfio-pci error -22

Can anyone point me to a list of what the vfio-pci error codes mean? I’m getting a error -22 when attempting to bind any video card to vfio outside of slot 4 on my motherboard. The error pops up in the kernel log / dmesg. In desperation I installed Linux 5.0.5 fc29 x86_64 with the acs patch applied, and tried all the acs override combinations. I did notice that all slots except 4 were putting GPUs in an iommu group with a bunch of other stuff. Slot 4 seems to be the only isolated slot without acs override. Even when the override split everything into its own group still getting -22. Confusingly there’s posting of my board doing fine but from 2+ years ago. So I wondered if something changed in the bios, the CPU or Linux kernel? Board is a rev. 1.0 gigabyte designare ex x399, CPU is the Ryzen 2950x and like I said, Linux 5.0.5 (acs patch) currently also tried 5.0.6 (no acs) The board is new old stock as far as I can tell? I did have to update the bios for it to detect the CPU.

For whatever reason on my Gigabyte, none of the PCI devices with an address 4x:xx are assigned to an IOMMU group, which includes the first two pcie slots and the m.2 slot