VFIO/Passthrough in 2023 - Call to Arms

Wikis & How-to Guides
21

As much as we would love to make it this simple, there are an absolute ton of edge cases that prevent this becoming a reality. For example:

  • Number of CPU cores available
  • Core numbering differences depending on CPU and/or microcode versions (breaks pinning, etc)
  • Brand of CPU (AMD vs Intel)
  • Brand of GPU for Guest usage
  • iGPU vs dGPU for the host
  • Laptop users with muxed vs muxless hardware
  • Storage medium (block device, passthrough, qcow, iscsi)
  • Windows 11 TPM configuration
  • Problematic passthrough GPUs
    • Laptop dGPUs
    • AMD GPUs
  • Problematic motherboards
    • Ones that have very poor IOMMU grouping requiring ACS patching the kernel
    • Ones that just refuse to properly work for VFIO at all for unknown reasons.
  • QEMU/OVMF 64-bit bug with IVSHMEM (this is a recent issue)
  • Audio configuration (though this is easier since LG B6)

I am sure there are many things here I am missing. There is no way to simply make a single guide to rule them all, there are too many possible configurations which cause too many edge case issues to document.

2 Likes
22

The old method of hiding the KVM leaf and setting a random hv-vendor-id is all that is needed. You actually still need to do this anyway for consumer cards if you don’t want to cripple the custom resolution feature.

2 Likes
23

You may be interesting in in quickemu, which is the absolute simplest way to get wide variety of VMs (linux, windows, even macOS) up and running.
https://github.com/quickemu-project/quickemu.

A nice visual overview of how simple and quick it is:

For the reasons gnif described, it doesn’t currently handle GPU passthrough. That said, there is one guy exploring adding the functionality as a related project:
https://github.com/quickemu-project/quickemu/issues/688
https://github.com/HikariKnight/QuickPassthrough

2 Likes
24

For me VFIO has been working quite alright ever since I built a home server back when the very first ryzen (1700) chips hit the market.

One of my goals was to have a headless host machine. The machine has no graphics output at all for the linux host, and only passes through the Radeon R5 230 (yes, it predates even the Polaris architecture.)

This seemed to be a major pain point to many back in the day but I could make work quite easily (with MSI B350 Mortar Arctic). I had to resort to compiling the pci-stub module into the kernel to ensure that it loads before anything else can grab the GPU, but that’s pretty much it. I have successfully used this with Windows, Linux, BSD, etc. guests. If I had to specify one annoyance, it is that the very first time the host boots, the GPU stays in a powered on state (that is, it outputs a black screen, because the firmware has initialized it) until I run and power off a guest VM that knows how to “shut down” the GPU.

What worked much less satisfactorily to me was attaching… other stuff… to the guests. I found plenty of situations where not passing through an entire USB hub (and attaching individual USB devices to the VMs) would not work quite right. Xbox controllers come to mind as an example: Windows makes the connected Xbox controllers reset for some dogforsaken reason. That then detaches it from the host (and thus VM) until the controller restarts. Few moments later the host notices a new USB device and handshakes with it, it gets connected to the VM again, Windows resets the controller…

And since I do not use VMs all that frequently, pci-stub-ing all of my best (built-in) USB ports on the machine was quite a hard pill to swallow just to make a controller work. It seems that if you’re thinking of using USB devices with both host and guest systems, you should be looking at a hub<->port layout that makes sense to you (or plan for a USB add-in card.)

25

Nowadays, SSD makes more sense now. I would love to learn how to put 10 ssd drives in to a mainstream PC.

26

Still not working.
Even with this script.

27

I wrote down my experience here:

Something broke my PCIe passthrough.
The VFIO unbinding to be precise.
It worked before but now it doesn’t and every time I get it to work it starts breaking again.
Feels a bit like someone is behind it.
I got it to work once when I turned FireFox off, but not a second time.
The UEFI shell is missing again in my BIOS but I did not change anything except an update for something that looked like java script.

  • MSI X570 A PRO 128GiB RAM
  • RYZEN 5950X
  • AMD Radeon HD5400 passively cooled host
  • AMD Radeon RX590 Windows guest

I use looking-glas and hope that they fix AMD issue so it works without a connected screen.

When my neck is killing me I use a beamer for the ceiling.
I built a frame so it can work in any orientation.

My second system broke this year.
The motherboard or CPU seem to be broken.

If I win the giveaway please only AMD parts.

Wish me luck for the lottery so I can buy the stuff myself :wink:

28

I’ve had trouble with latest kernel and vfio driver, but being on Ubuntu I had to read and come up with my own solution because no one uses Ubuntu and passthrough anymore!

It’s super simple; Instead of vfio I just pci-stub the devices on kernel boot params I want to passthru and then hand them over to vfio driver.

29

Same issue here with the latest Ubuntu 23.04.
I would love to learn how to do that.

30

6.0 or 6.1 kernel “broke” the vfio-pci.ids module parameter/command-line argument. Fortunately the Ubuntu initramfs scripts are pretty smart and you can get it working again fairly easily. For each device you want to bind to vfio-pci at boot, you have to add a softdep for its default driver. update-initramfs will parse this and load vfio-pci first (and parse and bind the list of IDs before the other driver has a chance to bind it).

Here’s my /etc/modprobe.d/vfio.conf for two Samsung NVMe drives and an NVIDIA audio device on Ubuntu 23.04 with 6.4.6 kernel:

$ cat /etc/modprobe.d/vfio.conf 
softdep nvme pre: vfio-pci
softdep snd-hda-intel pre: vfio-pci
options vfio-pci disable_vga=1 ids=10de:1aef,144d:a808,144d:a80c

And just for the sake of completeness, run this command to update initramfs for all installed kernels after making any changes:

$ sudo update-initramfs -k all -u

That got everything working again. HTH.

N.B. I use driverctl for the actual GPU device itself so I can un-bind it as necessary.

2 Likes
31

Just using any in the full line up Quadro card , it’s a plug and play directly pass alltime working. just 6-7 setting to add and full system do pass. Esxi do work good and give best latency.

32

IMG_0813
IMG_0813900×1200 330 KB

Here’s a photo of the computer on my test bench (an IKEA cabinet door). X670E ProArt, 7950X, 4xKingston Server Premier 32GB ECC RAM, ASUS TUF 6900XT on a riser, four SN850 2TB SSDs, currently no SATA drives, 1000W PSU. 140mm water cooling loop with 3000RPM Noctua fans.

3 Likes
33

Running a Supermicro M12SWA-TF motherboard with a 5975WX CPU using Proxmox.

Was able to passthrough the following GPU’s.

  • Gigabyte GeForce RTX 3060 WINDFORCE OC 12GB, Model number GV-N3060WF2OC-12GD
  • EVGA GeForce RTX 3060 XC LHR Graphics Card 12GB, Model number 12G-P5-3657-KR
  • Nvidia RTX A4000

Could not get the XFX Radeon RX6700 SWFT 309, Model number RX-67XLKW working, seems to be effected by Reset error and others have reported the same trouble.

Was unable to get any GPU working in Slot 7 of the motherboard (closest to the CPU) it just gave GFX card related errors around FLR reset (see below) which cause the VM to fail to start up.

Jan 11 14:51:38 pve kernel: [83097.716315] vfio-pci 0000:41:00.0: not ready 1023ms after FLR; waiting
Jan 11 14:51:39 pve kernel: [83098.771575] vfio-pci 0000:41:00.0: not ready 2047ms after FLR; waiting
Jan 11 14:51:41 pve kernel: [83101.011621] vfio-pci 0000:41:00.0: not ready 4095ms after FLR; waiting

My setup uses three RTX A4000’s GPUs.

When the system is under high IO load I get audio crackling and sound lag when watching videos/youtube. Using CPU core pinning and NUMA node assignment improved the audio, but it still does lag and crackle a bit.

Still need to try using a virtual sound device in the VM and streaming it to the Host via Pulseaudio to see if that works around the issue.

34

I saw the guide was only about AMD GPU.
I have a system with both a AMD gpu for linux, and a Nvidia GPU i wanted to pass through.
I’ve seen a lot of “single GPU passthrough” guide and “reset issue” only on AMD so i guessed that i could somehow use the nvidia GPU for my 4th display port display 80% of the time, and when time came unmount it and pass it through to a VM.

Sadly i can’t unload the nvidia module , even after killing the GUI :frowning:
I couldn’t find a definitive “it’s not possible”, can someone give me one ?
I would like to avoid losing a screen and blacklist the card at boot for the 1h a week i’m gaming.

35

I started with VFIO back in 2019. I used regular Ryzen desktop processors like the 2700X with Gigabyte motherboard. The idea was that Gigabyte offers an option in the UEFI to select the GPU slot that should be initiated first. However, after a lot of back and forth, it turns out that option does not really work. It does work as long as the GPU you select to be initiated first is the only one with a monitor attached to it. In case another card, in any PCIe slot that’s position physically above the one you want to initialize first, has a monitor connected as well the option won’t do what it promises. In most cases, with most UEFI versions, even Gigabyte board will initialize the first card with a connected monitor. I was never able to make this UEFI option reliably do what it promises.

As a result I decided to place the GPU for the host in the primary PCIe slot and the passthrough card in one of the lover PCIe slots. This did reliably work on any mainboard I had since then, but it is a bit of a pain since the layout on most consumer boards want you to place the GPU in the first slot and you often run into trouble when you have a need to use more add-in cards.

At some point I switched from regular Ryzen to 3rd generation Threadripper PRO. I still need to place the host card in above the passthrough card, but at least on this platform I have plenty of slots.

I use passthrough mostly in combination with Looking Glass to be able to play some Windows games, the only other virtual machine that uses my GPU is the one I use for transcoding videos.

Looking Glass works perfectly fine in combination with my 42" inch, 120Hz, 4K screen. Thanks for that to @gnif.

Maybe a short overview of my systems specs:

  • Threadripper Pro 3995WX
  • Asus Sage WRX80
  • RX 550 2G for the host
  • RTX 4080 for the guest

That’s not true, I am on a 6.1 kernel and it worked completely fine the entire time. I think, Ubuntu somehow broke it in their compiled version of the kernel for whatever reason, is the correct statement here.

1 Like
36

Fair point. Or possibly in their initramfs scripts.

37

I have a mixed bag of success and things to work on. Things are mostly working and I don’t stress out over VFIO/GPU passthrough.

Desktop:

  • ryzen 5 3600 CPU
  • 6900 xt challenger GPU
  • gtx 750 ti for passthrough
  • looking glass B6
  • debian 12 host
  • win 10 guest

This works fairly well, but I have fairly pedestrian requirements. I mainly use the win 10 guest for filing taxes and some job interviews that use teams. You could make the case that I don’t need GPU passthrough for this.

On debian 12, the gtx 750 ti passthrough GPU uses the VFIO driver, but if I recall correctly, on debian 11, it would use the nouveau driver, and the VFIO would take over when I started the virtual machine. I couldn’t get this to work on debian 12 the same way it did on debian 11, using the same hardware.

Another interesting feature is that I don’t need an EDID dongle or HDMI plug to use the 750 ti?? Win 10 seems to detect a virtual display and uses it. This makes it easier to boot the system and be able to see output on the 6900 xt which actually connected to monitors.

Also not working in this set up is passthrough for a rx580 GPU. Well, the passthrough works, but I could never get the driver to reset, even with the vendor reset kernel module. After shutting down the VM, the fans would spin just loud enough to be annoying. I would rather use this newer GPU, but the old GTX 750 ti 2GB works for my needs.

Laptop:

  • intel 10th gen, 6 core something?, intel iGPU
  • looking glass B6
  • debian 12 host
  • win 10 guest

This works a peach too and I have no complaints other than using the win10 VM makes the fans spin.

This setup uses the single GPU and Intel GVT-g to virtualize the GPU. I have a qemu hook that creates the MDEV virtual device when the VM starts up. There is no EDID dongle/HDMI plug for this setup either.

I use this win10 virtual machine for zoom presentations and it has been mostly seemless. I do have some occasional issues with connecting/passthrough the external audio/video to the VM for the zoom conference, but that might be the room hardware and not the computer/passthrough setup itself.

Shout out to @gnif for Looking Glass- I use it with all of my passthrough setups and it makes the experience much easier since I don’t need a KVM to get to the guests (but I do have a KVM for a physical work PC).

1 Like
38

That is so cool, I wanted to use NixOS on the same motherboard similarly. Could I check out your configuration.nix?

39

For me, the VFIO with VMs is an awesome way to use an OS I prefer (Fedora) at work. Unfortunately, some software I need for design, so 3D heavy, requires Windows - wine could be used for all of them with great success, (I tested it before), but I use some USB peripherals that do not work with Wine. So VFIO it is. Yes, I use VFIO at work daily :slight_smile:

Important distinction from most (if not all) of the guides, I do not run my VMs as root. I run them as my regular user, which forces some limitations. As far as I know you still can’t use virtiofs to share directories with the host, but for me this is a non-issue, the passed-through devices must be owned by your user (which is taken care of by udev config) and I needed to create an alternative to vfio-isolate, which you can find on my gitlab (I can’t paste links here - just replace the space with dot: gitlab com/krokodyl1220/vfio-vm-tools). This simple script handles CPU isolation and allows the VMs to have great performance.

My work PC config is:

  • Ryzen 5700G
  • Gigabyte Aorus B450 Elite
  • 64G of RAM
  • Nvidia GTX 3050 for Windows guest (in the primary GPU slot)
  • Radeon RX550 for Linux guest (using NVME → PCIE converter)

I also pass two onboard USB controllers, one to Windows and one to Fedora, and that allows me to have dedicated USB hubs for both VMs. On my motherboard the primary GPU slot is in the separate IOMMU group, the same as the primary NVMe slot, so these were the sensible choices for GPU connection.

My host OS is OpenSuse Leap 15.4, but any relatively stable distro will do (such as Alma, Debian or something similar). In the past I used Arch but I would not dare to do it today :smiley:

I use two VMs daily:

  • Windows 10 - for a few pieces of SW that need it
  • Fedora - for all the rest of the work.

This approach allows me to use a fast-changing distro such as Fedora for having fresh packages without having to worry about the updates breaking the GPU passthrough. I also run a few VMs inside my Fedora VM, so nested virtualization also works great.

For networking I setup 2 network bridges on my host, each as a separate subnet - one is for direct communication between Linux VM and Windows VM (Linux acts as a SMB server, hosting design files accessed by my Windows software), and one is setup between Linux VM and host, to allow the Linux VM to access Windows guest SPICE for keyboard and mouse. For connecting to the Windows guest I use Looking Glass from my Linux VM, and I pass my peripherals between my host and Linux guest using evdev. Thanks to this I can use a single cable for each peripheral and can very quickly switch if I need to access a host. One limitation - evdev doesn’t handle keyboard indicators, so capslock and other indicators on regular keyboards do not work.

This setup is rock solid, has saved me from fixing my systems a few times (if something goes wrong with the guest - just restore the snapshot and you are good to go) and allows me to use Windows software feeling as if I was using Linux :smiley:

4 Likes
40

You know, that’s actually something I’ve somehow never considered, is the permissions of VMs. I’ll have to look into what’s actually going on when I fire one up.

Here’s the repo you mentioned as a full link: https://gitlab.com/krokodyl1220/vfio-vm-tools

Good work. One thing you should consider is adjusting the readme with a summary of what you changed, so people will know it’s not just a straight clone.

Thank you for the writeup!

1 Like