Return to Level1Techs.com

VFIO AMD ThreadRipper 2019 Notes ( Near Bare Metal Performance )

#1

These are my notes to get AMD ThreadRipper up and running after VFIO setup. I have added Arch PKGBUILD files to build on the latest kernel. I have also added a start script I use.

The biggest tip is to use “Virtual network sound card for Microsoft Windows” that uses ivshmem. Very low latency. Maybe we could add this code to looking-glass.

VFIO AMD ThreadRipper 2019 Notes ( Near Bare Metal Performance )

4 Likes

#2

Thanks for sharing your notes! Looks great! I should post mine to compare.

Out of scope.

Definitely gonna check that out though.

1 Like

#3

So tempting to try again but my x399 motherboard is a dud. Most of my slots don’t support IOMMU for some reason.

Maybe next year’s TR update from AMd will bring new boards to the field and I can upgrade.

1 Like

#4

What board do you have?

How is that possible?

1 Like

#5

Giga Designare x399 … The first two slots off the cpu, PCIEX16_1 and PCIEX8_1 just don’t show up in any iommu groups, and their pci bus addresses are “weird” compared to cards in the rest of the slots.

I posted about it a while back, but gave up after getting frustrated and just wanting the rig working. I was able to pass-through a card in PCIEX16_2 but using that slot is not ideal for the layout I want.

post here

1 Like

#6

So glad I didn’t buy that board then. :frowning:

2 Likes

#7

Umm I don’t think you’re doing something correctly as this thread on unraid shows slots 1 and 4 work: Gigabyte X399 Designare EX Threadripper

1 Like

#8

Just bad silicon I think for me.

The pcie addresses are different in that post. All of mine start with a 42:0:0 instead of 0:0:0 … I found a Linux kernel dev mailing list that explained what is happening but it’s over my head.

1 Like

#9

Any Benchmarks ran inside the Windows VM?

1 Like

#10

feeling masochistic, I booted the latest unraid and wiped the drives on my ripper. GTX 1080 in slot 1, acs override on, vfio-pci.ids excluded … try to pass through, vm hangs at boot. same as it ever was unfortunately.

1 Like

#11

Any chance of walking through and diagnosing the problem with you?

  1. What BIOS are you on?
  2. Have you enabled SVM, ACS, and IOMMU in the BIOS?
  3. What kernel revision are you using? Has it been patched for ACS override?
  4. What boot/kernel flags are you using? (you need at least amd_iommu=on iommu=pt)
  5. What is the output of dmesg | grep -i -e DMAR -e IOMMU and dmesg | grep AMD-Vi?
  6. What is the output of lspci -tv?
  7. What is the out out of lspci -vv for your GPUs?
  8. What does find /sys/kernel/iommu_groups/ -type l output?
  9. Can you provide your guest XML file?

Logs from dmesg when you run your vm are also appreciated.

Feel free to start your own thread with this info, and I’m happy to run through troubleshooting with you. I would however recommend trying the latest Pop!_OS guide posted by Wendell, as that is fairly helpful.

1 Like

#12

Thanks for this ! 2950x user here on MEG Creation.

I mostly got a near ideal setup, but there was some pointers in here that were very helpful. I’m using Arch’s Linux 5.2 atm and it’s not playing nice. I’m going to give your kernel a try

1 Like

#13

Thanks for the offer. I gave up pretty quick on this. I copied my config from previous attempt with unraid.

2 Likes