VFIO/Passthrough in 2023 - Call to Arms

Communities Events
1

> Enter The PC Giveaway Here! <

I am working on updated guides for VFIO in 2023 and I want to know what has and has not worked for you in your build.

The documentation that exists out there on the internet has become a bit dated and I am going to try to organize and connect the dots. The reddit vfio community is very active, but I am confident I have more permanence of information on the intenret than reddit and other vfio communities that have sprung up.

The VFIO 2023 Project

The first thing iis going to be a giveaway and build video, thanks to AMD and Corsair.

It turns out that the Radeon 7000 gpus mostly DO support vfio, but there may be some rough edges around certain motherboard/bios/agesa combo and 7000 series gpus (whereas 6000 series can work fine).

This is obfuscated a bit because you can get the card into an impossible reset situation when resize bar windows are off (there are two, and it seems important that the two bar windows are kept in sync).

I have been using the threadripper pro system I have FOR YEARS (with unupgraded gpus) and I still love it. I see a lot of enthusiasm here, but I’d like to do quick guides like I used to that really cut through.

VFIO Quality of Life - Looking Glass and more

Secondarily, there is a whole software ecosystem to support this use case. Pipewire, some new exciting stuff from @gnif and gold old standbys like gnif’s looking glass… also need a whole suite of howtos.

Building curated VFIO community resources

I’m either going to write them, or link them.

So, as succinctly as you can, share your vfio success story or failure. Be aware I may split off your replies about your build into separate threads as this thread evolves, but your stories will be linked to from here.

Finally, sr-iov on the intel arc a770, whether officially or unofficially is on the horizon. I am thinking I can lay out what needs to be done and setup a bounty to connect the dots here. This is already a thing in china with the A770… more details on that soon (and separate thread).

More Soon ™

36 Likes
2

I guess you want something like this?

I’m using a 7950X on an X670E ProArt with a 6900XT. My monitor is a TV plugged into one of the USB-C ports on the motherboard. The iGPU renders my desktop and non-3D applications, while the dGPU renders the 3D stuff whenever it isn’t being passed through.

I pass through the 6900XT to a virtual machine and then connect to that with RDP, which lets software like Solidworks use it to accelerate their graphics even with nothing plugged into the GPU. I can also plug the GPU into the TV with a HDMI cable and use looking-glass, which I usually only do when I want to play a video game that won’t run on Linux (because the games I’ve tried to do this with won’t work under RDP).

This setup works fine maybe 25% of the time. I can always start the VM just fine, my linux desktop stays active and any software launched after the VM gets the GPU will render on the iGPU without issues. However I suffer from the reset bug, and 75% of the time shutting down the VM won’t return the GPU to Linux and I have to reboot to fix that.

I’m quite satisfied with this setup. The onboard graphics through display port will do 4k120Hz with chroma 444 on my TV just fine even though the TV has no display ports (it’s connected with a passive USB-C → HDMI cable) and the performance penalty for using the dGPU without anything plugged into it appears minimal. Looking-glass also works fine. The X670E ProArt has good IOMMU groupings, even though I’m bifurcating the x16 slot into x8/x4/x4 each connected device gets its own group, and the same goes for the M.2 slots on the motherboard itself. All the NICs however share a group, so if you want to do passthrough networking you’ll probably have to bridge a virtual interface with a real one instead of just passing the real one through. The USB controllers are also all lumped into this group, so you won’t pass through a specific set of USB ports either (I’ve tried doing this for the motherboard headers, nothing works). Also, for some reason the 2.1 headers on the motherboard share a bus in lsusb with some of the 3.1 ports on the rear IO, so I can’t pass through a root hub either.

6 Likes
3

I’ve dropped this before, but I’ll do it again here. It’s basically the most comprehensive “the state of resource isolation in regards to VM’s and Proxmox” that I’ve seen, and is still a good reference for other distros. Especially important with multi-numa processors becoming a thing even on desktop with e-cores being a thing now.

https://forum.proxmox.com/threads/hey-proxmox-community-lets-talk-about-resources-isolation.124256/

As far as my own systems, oof… it’s a tangled undocumented mess there’s no way I reproduce, which has seen issues come and go, and who knows what else I’ve broken somehow. I’m in the middle trying to grok NixOS, first on my laptop, then on my NAS/VM machine.

In a few months I’ll hopefully have a configuration that’s essentially documented simply because it exists, and all I’ll need to do is hand out a git link (which will include a note on my hardware and bios/uefi settings) it’ll be 100% and immediately reproducible to anyone else with the same hardware.

That said, this thread is a good reminder that I should also make a note of what doesn’t work and when. And perhaps the versions of packages/kernel that broke things

1 Like
4

pics too would be awesome! Sorry it only works 25% of the time. Weird that its with the 6900xt.

can you check the bios settings linked above? any difference? I dont wanna buy yet another mobo. LOL

1 Like
5

I have lots of notes from my two daily drivers, an X99 rig with a passed-through 3080 and a X570 rig with a passed-through 3090. I will start to document them. I use a KVM for both so no Looking Glass. I also have it set up so I can shut down the VM and use CUDA from Linux.

I left some notes on your linked post about x-vga, re-bar, and mainline. Any modern VFIO guide also needs to cover driverctl and vfio-isolate (I use the /etc/libvirt/hooks/qemu example script almost verbatim.

4 Likes
6

My build is documented here:

  • Asrock Rack ROMED6U-2L2T
  • EPYC 7402P 24-Core Processor
  • Noctua NH-U9 TR4-SP3 Air Cooler
  • 128GB ECC Unbranded DRAM 2x32GB, 4x16GB, the ROMED6U has 6 dimm slots)
  • Proxmox VE as main OS
  • NVidia 3070TI passed through my main Windows WM (for Gaming)
  • Sapphire GPRO e9260 8 GB passed through my main OSX VM (for Working)
  • 2x Fresco Logic FL1100 USB 3.0 Host Controllers, passed through either my main VMs, connected using two of the ROMED6U Slimline connectors to external PCIe 8x slots

I have gone the route of 2PC on 1 host full tilt, each VM has a dedicated GPU->Displayport->Input on a 43" Display (used to be a Philips BCMxx, now an LG)
The server is in my attic, video and USB are transmitted using HDBaseT Extenders

I also am using 2 keybords/2 mouses/2 audio interfaces, two fully separated machines
That was my setup until 3 months ago when I finally gave up on OSX-Hackintosh and had my company buy me a mac mini
I have then removed the Sapphire GPU and the USB card, swapped in a 4xNVME card and shuffled my other stuff around to promote the Epyc machine to forbidden router duty as well using Vyos
The reason for moving away from OSX on KVM had nothing to do with KVM, a lot to do on the poor efficiency of OSX running on an Epyc machine and legacy GPU …

I am still using the windows VM, and in order to get decent performance out of it I had to implement shielding of the VM cores, pinning as well and I went the full monty and pinned the interrupts for the GPU and USB controller to the vm cores as well
I adapted an existing set of proxmox helper scripts to implement pinning,interrupts masking, using a custom governor and made them available on github:

I can start and stop my VMs from my Stream Deck, remotely attached to a raspberry pi, the streamdeck has also commands to start/stop the server using ipmi commands, and I can do that with custom keypresses on a 4x4 mini pad as well from the bottom of my desk …

IMG20230727231439
IMG202307272314391920×860 149 KB

IMG20230727231553
IMG202307272315531920×860 62.3 KB

Audio setup is a bit convoluted, but I like my virtual audio channels to map to an interface with actual sliders, so I am using a LiveTrak L-12 as an output to speakers-wired headet-bluetooth headset, and using an XLR mic as input, the Windows Audio interface feeds into two channels of the main interface as well, as well as the kids piano and a couple of othe occasional inputs…

16904927985101068990220358868993
169049279851010689902203588689931920×1440 323 KB

15 Likes
7

I have a very similar setup. What is your bare metal OS? Do I understand you correctly that your 6900XT renders Linux-compatible games and displays out via the TB4 port of your motherboard (outside of a VM pass-though)? Did you need to connect to the DisplayPort in of the motherboard with this configuration?

8

Sure, I can take any picture you want when I get home next week. Do you just want one of the components assembled on a test bench?

I should also mention, that before I RMAed my B650I Aorus Ultra I did passthrough with it, too. Same setup, x16 bifurcated into x8/x4/x4 where the x8 device gets passed through. IOMMU grouping was fine there too, each PCIe and M.2 device got its own group. It also separated USB controllers, which was super convenient. I could plug a PCI bracket with four USB ports into the motherboard headers and pass through the controller doing the internal ports to have a few ports that would automatically pass through to the VM. It didn’t like my USB-C → HDMI cable though so I ended up using an active displayport → HDMI adapter instead.

As far as AM5 goes I wholeheartedly believe the X670E ProArt and the B650I Aorus Ultra are the best motherboards available. The ProArt supports ECC explicitly and I’ve seen it log an error, so it works, and the Aorus Ultra would at least boot with ECC RAM even if I never did try it long enough to see an error (I need about 100GB of RAM for work so had to use non-ECC 96GB kit). The ProArt overclocks memory a lot better than the Aorus, which you wouldn’t expect comparing a two DIMM per channel board to a one DIMM per channel board.

The only BIOS option I see mentioned in the other thread is ReBAR? I’ve been running with ReBAR off the entire time. I just assumed that was a prerequisite to get passthrough to work at all, and having it off doesn’t seem to majorly affect performance on the 6900XT. I can’t see any other BIOS options mentioned? You may need to hold my hand, I’ll eagerly test things on my hardware for you but I’m nowhere near your level of skill.

1 Like
9

Bare metal OS is NixOS. Yeah, in Linux the dGPU renders games on the iGPU output, which is displayport over USB-C. I don’t have TB, it’s an AMD board, but yeah, it’s basically TB (USB4). The motherboard has a displayport input which I can connect to the dGPU to get the dGPU output direct from the motherboard, but I don’t do that. I use the internal graphics on the 7950X to render the desktop, the dGPU is idle most of the time. dGPU PPT is 5W on the desktop, iGPU sits on ~45W on the desktop. If I start a game, dGPU hits 312W instantly, iGPU remains ~45W. I’m using a shader wallpaper plugin so power use is a bit high because of that, but iGPU easily keeps up with a 4k120Hz monitor despite it. I wouldn’t trust these numbers other than broadly though, sensors on AMD are always really bad in my experience.

Not all games automatically detect the dGPU and render to it, so them I will use a Steam launch option for.
DRI_PRIME=pci-0000_03_00_0 %command%
Where pci-0000_03_00_0 is my 6900XT.

3 Likes
10

I have some type of VFIO on all my 3 main computers:

WORKSTATION: Asus B650 ProArt motherboard with 7900x cpu. GTX1080ti passed through to a Windows 10 VM using KVM/Virt-Manager for CAD work. I have Lookingglass setup, but usually just use different inputs on my dual monitor setup as Lookingglass does not (did not???) support dual monitors. Keyboard and mouse via evdev. Host OS Manjaro XFCE with an AMD RX470 for video duties. I also pass through the same 1080ti to an ubuntu VM for running stable diffusion. Obviously the two VMs are never run at the same time. Very stable so far (3 months since I built it.)

SERVER: Supermicro X10SLR-F with a E5-2696v3 and 128GB RAM. This is a proxmox host where I passthrough a quadro card to a Jellyfin VM. I also pass an HBA to a TrueNAS Scale VM. This is very straightforward and stable; the proxmox vfio guide worked like a charm.

LAPTOP: Legion 15 with an 5800H and a RTX3060 MaxQ. Host Manjaro XFCE. I pass the mobile 3060 to a Windows VM for CAD on the road (again KVM/Virt-manager) and use Lookingglass to great effect. The radeon graphics of the 5800H is used for daily driving the Manjaro desktop. This took a little bit to get working. I had to hide the KVM / other basic tricks to get past error 43, but that bit was pretty standard solves. I also had to pass an acpi table to the VM because of the host’s battery created issues with the GPU in the VM. Once all that was sorted it has been stable for 2 years now.

4 Likes
11

Sneak Preview :smiley:

2023-07-28-153905_1557x1259_scrot
2023-07-28-153905_1557x1259_scrot1557×1259 112 KB

5 Likes
12

I feel like the uninitiated would want something indepth explaining the nuances. At least for the types that learn that way. I tried it once on a laptop. The experience was interesting. Ive only got experience with nvidia. I passed through the 980ti once but the experience wasnt great. Not vfio or looking glasses fault. I was the idiot doing this in a hardened arch build.

But yeah this project is ultimately the coolest if not one if the coolest projects ive seen on this forum. Man to get away from dualboot would be bliss. Particularly on a laptop. My lenovo legion exposes everything. I wonder if I should try to wrangle its 3070 into submission for the task and have debian use the integrated radeon. This could go horribly wrong depending on which gpu trully controls the output to the eDP screen

13

Is single gpu multiple passthrough a thing yet. As in can host and vm use the same gpu or is that work still far off?

1 Like
14

That’s called SR-IOV. It’s a supported feature in high-end professional cards like Radeon Pro and Quadros, but people have had it working with normal consumer Nvidia cards as well. No such luck with consumer AMD cards unfortunately.

3 Likes
15

Hell yes. Im all in on nvidia anyways

16

I wonder the latest status quo on AVIC. Is it dead or someone lucky got it working in 2023? I believe it’ll be a great addition to VFIO/passthrough on AMD platforms.

The thread I linked just came up on a quick search. I think I saw an earlier and longer thread as well as a long Reddit thread by the same author. I was able to get AVIC working in Linux 5.y.x (what exactly I forgot). After a couple of Linux releases, it no longer worked.

Few folks seem to care about it on Internet. I meant isn’t it next to the greatest thing in PCIe passthrough for hyper scalers in the cloud? But seems so little discussion about it.

17

Been using it for about 8 months now, no issues. Some hardware doesn’t play nice with it, but it’s working for the most part.

1 Like
18

I am still using a dated i9 9900K with a Geforce 3080ti and a Radeon RX 6600 with Proxmox. I realise, that I am still pretty much a Linux newbie and don’t know much about the inner workings and most of the various linux CLIs commands - but I am very happy with my daily driver I build.

What didn’t work:

  • Doing vfio in a Desktop distro (Arch / Manjaro), to much latency, to many workarounds (hugepages setup, core seperation / pinning, the need to recomple kernels after a change added 2min boot time with VMs which have more than 16GB RAM) and a lot of other issues which required constant mantainance
  • passthrough of legacy hardware (Windows XP, Win7 with older non-UEFI GPU cards), seabios passthrough - my goal was to build a retro PC based on (more) modern hardware (Intel i7 8700K CPU and mainboard during that time), no HW / software or KVM solution could produce a successfull GPU passthrough > I am better off with PCem nowadays for these old games
  • HyperV nested virtualisation tanks performance in KVM

What worked:

  • 2 gamers, one PC setup using cheapish consumer hardware
  • using link files to get rid of unpredictable network interface names while moving PCie devices around (for example replacing a m.2 would isolate my Proxmox host because the network interface name wouldn’t match in the network config): NetworkInterfaceNames - Debian Wiki
  • ZFS - creating, maintaining and expanding pools is a breeze
  • passing though SATA controllers to use Windows storage spaces in a VM (workaround for a short time after importing my bare metal system back in 2021)
  • Proxmos NAS with various hardware passed through
  • no need to isolate HW with vfio-pci in Proxmox
  • Mac OS VM with GPU passthrough
  • the janky ass PCIe - USB (like) extenders which I also use to connect a 10Gbit network card to a 1x PCIe slot (hey, still faster than a vanilla 1Gbit connetion) and my USB 3.0 PCIe cards

Edit:

  • UEFI flashing a passed through LSI card (MINI-SAS) with OFW in a VM
  • Building a second storage PC with Proxmox which uses the flashed LSI controller

Here are older pics of my system

3181×2973 922 KB

4032×3024 1.2 MB

19

I guess this is a failure story.

If there is any takeaway I want @wendell or @gnif to have before I start my rants it is that I am going to ask on behalf of budding Linux enthusiasts to dumb it down even further, if possible, to a 1-click install or failing that, a step by step wizard tailored for people thinking about a Windows exodus.

IIRC I first encountered L1T during LTT’s first crossover episode, and that was about VFIO and Looking Glass! I remember seeing the Looking Glass page that day thinking “Okay! Lets get this thing started!” So I went to the Installation page and this greeted me on literally the first few lines:

What LG said

Installation

libvirt/QEMU configuration

This article assumes you already have a fully functional libvirt domain with PCI passthrough working.

If you use virt-manager, this guide also applies to you, since virt-manager uses libvirt as its back-end.

It assumed a lot on the abilities and information I do not have at the time. I didn’t know these words back then, and to be honest, I only have surface level understanding of what it exactly is even now: libvirt, virt-manager, QEMU, etc.

This intimidated me greatly and eventually dissuaded me at the time as I was just barely 6 months into Linux and I saw that this is waaaay over my head. I should have RTFMed more and I felt helpless, but I didn’t know what I didn’t know to move forward. Looking back, I wish there was an easy step by step wizard-like install for people like me who are experienced enough in Windows but not enough of Linux to pull my own weight.

Perhaps it doesn’t help that I have a preconceived notion of the ease of usability for VMs: VirtualBox. It was intuive and felt easy to explore on my own, vs what I didn’t know Looking Glass demanded of me.

Time and life went by and my use cases for LookingGlass has greatly diminished over time…

I don’t really play competitive online multiplayer games anymore, at least those that strictly require Windows for anti-cheat.

  • Decreased desire to play games and support the growth of the companies like EA, Activision-Blizzard, Ubisoft, Gearbox, Tencent subsidiaries etc and the like - the publisher of games that are actively hostile against Linux VMs.
  • Rampant cheating made online multiplayer competitive games less desirable.
    • Cheat software is now monetized and industrialized, guaranteeing a progressively sh*t experience for even the marginally popular online games and that only gets worse as it goes more popular.
  • A dedicated gaming PC isolated from the network seems to be a good way forward if you have the money to spare. Microsoft is also making sure that I have their telemetry in my games even if I had migrated away from their ecosystem anyway and I cant even escape my old identities to play Minecraft anonymously. At this point, how is owning a network segregated Xbox or a Playstation a lesser experience? Now I also have a PC for Adobe productivity stuff (for graphics design and layout).
  • Valve has also done a great deal of work to make Linux gaming more accessible, even the online competitive multiplayer: Planetside 2 comes to mind and I love that it works fine on Linux.
  • My reflexes aren’t the same anymore because I’m older or maybe I am just making excuses.

So I am suffering a loss of imagination as to why I need LG these days. I look to the community for inspiration when I see your cool use cases.

This is not to take away the good work @gnif has done. It is more than well appreciated and I may still have a use case for things in the future. Looking at Starfield right now

2 Likes
20

NVIDIA RTX 6000 ADA here. I hate it.

Apparently, passing through a consumer-grade GPU to a VM is a piece of cake, but the professional cards result in code 43. Lurking around on other forums, I’ve gathered that these professional cards can only be passed through via SR-IOV, and that requires a subscription license to do so.

So, NVIDIA’s reward for buying their most expensive graphics card is the privilege to pay more. :roll_eyes:

4 Likes