Return to Level1Techs.com

VFIO PCI Passthrough "works", but the VM is unusably slow

#1

It’s another one of these posts!

I’m trying to get a Windows VM working under Arch with VFIO. I’ve run into the issue that I’ve seen all over the place where the VM takes forever to boot and then runs incredibly slow (we’re talking 1 frame every 5 seconds), but none of the things I’ve tried have had any real impact. After some experimentation I assume this has at least something to do with the amount of memory I’m trying to give it (originally 16gb, down to 8gb), but none of the resolutions I’ve found online fix it.

Here’s my system:

Motherboard: Asus PRIME X399-A
CPU: AMD Threadripper 2950X
Memory: 64GB DDR4 (4x16GB)
GPU 1 (in use by Arch): Nvidia GTX 1060
GPU 2 (passed into guest): Nvidia RTX 2080

Here’s a gist with everything I could think that would be useful: https://gist.github.com/mcmillion/7cee49cdb9f27449c546accf822120fe

Here’s where I’m at:

  • AMD virtualization settings are enabled on the motherboard
  • I’ve also enabled ACS, IOMMU+IVRS, and set Memory Interleave to Channel as per https://tripleback.net/post/chasingvfioperformance/
  • AMD IOMMU is enabled in kernel options, as well allow_unsafe_interrupts
  • The 2080’s VGA, Audio, USB, and Serial Bus are all in a single group and passthrough to the VM fine (I get video output when I start it)
  • I’ve pinned the CPUs based on diagram from running lstop -l as per https://bytee.net/blog/amd-threadripper-kvm-windows-gpu-passthrough
  • I’ve set up hugepages and tried both the default 2048kb size and 1GB size
  • I’ve confirmed that info kvm shows KVM is running in the guest

This is my first time using QEMU/KVM and my first time attempting a VFIO setup, so I’m sure I’m missing something really basic.

Any help is greatly appreciated!

0 Likes

#2

can you give us your VM’s XML config file?

0 Likes