Return to Level1Techs.com

VFIO AMD ThreadRipper 2019 Notes ( Near Bare Metal Performance )

These are my notes to get AMD ThreadRipper up and running after VFIO setup. I have added Arch PKGBUILD files to build on the latest kernel. I have also added a start script I use.

The biggest tip is to use “Virtual network sound card for Microsoft Windows” that uses ivshmem. Very low latency. Maybe we could add this code to looking-glass.

VFIO AMD ThreadRipper 2019 Notes ( Near Bare Metal Performance )

5 Likes

Thanks for sharing your notes! Looks great! I should post mine to compare.

Out of scope.

Definitely gonna check that out though.

1 Like

So tempting to try again but my x399 motherboard is a dud. Most of my slots don’t support IOMMU for some reason.

Maybe next year’s TR update from AMd will bring new boards to the field and I can upgrade.

1 Like

What board do you have?

How is that possible?

1 Like

Giga Designare x399 … The first two slots off the cpu, PCIEX16_1 and PCIEX8_1 just don’t show up in any iommu groups, and their pci bus addresses are “weird” compared to cards in the rest of the slots.

I posted about it a while back, but gave up after getting frustrated and just wanting the rig working. I was able to pass-through a card in PCIEX16_2 but using that slot is not ideal for the layout I want.

post here

1 Like

So glad I didn’t buy that board then. :frowning:

2 Likes

Umm I don’t think you’re doing something correctly as this thread on unraid shows slots 1 and 4 work: Gigabyte X399 Designare EX Threadripper

1 Like

Just bad silicon I think for me.

The pcie addresses are different in that post. All of mine start with a 42:0:0 instead of 0:0:0 … I found a Linux kernel dev mailing list that explained what is happening but it’s over my head.

1 Like

Any Benchmarks ran inside the Windows VM?

1 Like

feeling masochistic, I booted the latest unraid and wiped the drives on my ripper. GTX 1080 in slot 1, acs override on, vfio-pci.ids excluded … try to pass through, vm hangs at boot. same as it ever was unfortunately.

1 Like

Any chance of walking through and diagnosing the problem with you?

  1. What BIOS are you on?
  2. Have you enabled SVM, ACS, and IOMMU in the BIOS?
  3. What kernel revision are you using? Has it been patched for ACS override?
  4. What boot/kernel flags are you using? (you need at least amd_iommu=on iommu=pt)
  5. What is the output of dmesg | grep -i -e DMAR -e IOMMU and dmesg | grep AMD-Vi?
  6. What is the output of lspci -tv?
  7. What is the out out of lspci -vv for your GPUs?
  8. What does find /sys/kernel/iommu_groups/ -type l output?
  9. Can you provide your guest XML file?

Logs from dmesg when you run your vm are also appreciated.

Feel free to start your own thread with this info, and I’m happy to run through troubleshooting with you. I would however recommend trying the latest Pop!_OS guide posted by Wendell, as that is fairly helpful.

1 Like

Thanks for this ! 2950x user here on MEG Creation.

I mostly got a near ideal setup, but there was some pointers in here that were very helpful. I’m using Arch’s Linux 5.2 atm and it’s not playing nice. I’m going to give your kernel a try

1 Like

Thanks for the offer. I gave up pretty quick on this. I copied my config from previous attempt with unraid.

2 Likes

Hmm, they do for me. F12e bios: https://pastebin.com/raw/kjc7BARn
1st X16 slot is group 11, 2nd X16 slot is group 2.

Here’s the debug info from months ago when I tried this under Fedora

For whatever reason on my Gigabyte, none of the PCI devices with an address 4x:xx are assigned to an IOMMU group, which includes the first two pcie slots and the m.2 slot