First VFIO build, lots of questions: single vs. dual GPU, virtualise Windows 10 vs everything, AMD vs Nvidia ...?

Software & Operating Systems VFIO
1

Windows 7 support is on its last legs and I won’t have Windows 10 (on bare metal at least) for privacy among other reasons, so Linux it is for our new desktops. High time, too – I’ve been running it on my home server, HTPC, laptop and occasionally desktop for 20 years, after all. The only reason we’re only doing this now is games, and between Proton and GPU pass-through that’s become a non-issue.

There are two gaming-related use-cases: 1) locally, ideally at 4K, 60 FPS, decent settings (I realise compromises will have to be made); 2) streamed to a HTPC+projector via Steam streamingm at 1080p, stable 60 FPS, maxed out settings. (The HTPC is running Linux and and has an antique Nvidia card, in case it matters.)

Problem 1:
The Windows VM is a band-aid for me, I’d much rather run everything that works under Linux, so in a dual GPU setup I’d need two capable GPUs, which is expensive, hard to cool & loud, but only ever use one at a time.
Could I get by with one GPU, switching it between guest and host? I’m guessing that would be like dual-booting in terms of having to shut everything down, so, no.
Could I use one strong and one weak GPU and switch those around? Basically only give the strong one to Windows for games that require both Windows and maximum GPU power? We mostly play one game at a time, so switches won’t be very frequent.
Would it be better to just virtualise everything, e.g. Proxmox on the metal, Linux and desktop VM(s) on top of that, passing the GPU to each one (and obviously only running one at the same time)? Does that even work? I.e, save state of work VM and shut it down, start gaming VM, play, shut that down and resume the work VM where I left off?

Problem 2:
I’ve been using Nvidia cards under Linux for the last couple of years, because their proprietary drivers just worked, for the most part, but I’d prefer AMDs much more open approach. I’d also prefer a vendor that supports my (pass-through) use-case (AMD) over one that actively sabotages it (Nvidia). Stealthing the VM might stop working at any moment, and it supposedly also disables various Windows optimisations for VMs. Clear-cut, then, were it not for the infamous “reset bug” that reportedly still plagues the VII, even Navi …
So, which GPU brands or combination of brands is currently recommended?

(The rest of the build is a bit vague, yet: 3[789]00X, decent X570 board, 2*16 GB RAM.)

1 Like
2

I am having pretty much the same questions. Looking at https://passthroughpo.st/vfio-increments/ a cheap host gpu seem to be enough? The GT710 does not seem to support 4K output above 60Hz though?
To add a question, can the host use the more powerful GPU and then switch to the weaker one when running the VM?

3

Far from an expert here but as I understand you can’t switch GPU’s on the fly as you have to reserve the GPU you’re passing with the VFIO pci stub driver so that the kernel doesn’t load a driver for it. Even if you could remove the VFIO driver and load the linux driver you probably couldn’t pass the GPU to the guest again until you rebooted as the VFIO driver can only be loaded at boot. Not ideal.

Any solution that requires booting into another distro to switch GPU’s feels like defeating the purpose a bit to me as then you’re effectively dual booting (though arguably w/o Win10 on baremetal).

It’s too bad that SR-IOV graphics support in linux sounds like it’s not totally baked yet. Then you could share a single GPU between host and guest by dynamically moving GPU resources between host and guests.

4

I’m also considering a budget VFIO build, for me I have decided to go with a 3400G, RX 580 and a B450 motherboard. I might reconsider and go with an RX 590 though if prices agree with me once I finally saved enough money. It is in the pipe, just not there yet.

The 3400G is a bit of a downgrade (about 5/6 the performance of the 3600), with only eight threads compared to the 12/16/16/24 provided by the higher tiers (3600, 3700X, 3800X, 3900X). However, this will still give you more than enough horsepower for Linux while you game in your VM in Windows. Another alternative is to keep your old nVidia card or buy an RX 550 card for your host.

If you go with Linux host / Windows guest, you should in theory be able to use all cores on the Linux machine plus the graphics card if the Windows guest is offline. If you go with the Proxmox solution, you will have 4 threads dedicated to Windows and 4 threads dedicated to Linux, and the same with the GPUs. Only working on second hand information here though.

Lastly, maybe you do not even need the VFIO for gaming. Linux can do pretty much everything these days with Steam+Proton, except play specific Windows titles. Check out your specific titles at protondb.com before deciding.

5

My understanding is that unbinding and rebinding drivers should work in theory. I’ve read reports that it doesn’t work with the proprietary Nvidia driver, because that refuses to unload once loaded, and that it does work with the in-kernel AMD one, in general. I’ve no idea if that applies to all AMD cards or even if rebinding support is enough for the actual use-case.

The idea is more like reconfiguring the system. I mostly play one game at a time. If that requires a strong GPU and Windows, I’d pass through the main one for the duration. Once it’s finished, I’d switch back. Even if that required a full reboot, it’d only be once every few weeks.

6

In that case you can probably accomplish what you want. Iirc there is a kernel parameter you can pass via grub to load the VFIO driver. Yup -

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#With_vfio-pci_built_into_the_kernel

So in theory you can just have an additional grub menu item. The only time you’ll have a problem is when you have two identical GPU’s and you need to use a script to tag it instead. One setup to pass the GPU and one that does not.

7

Thanks for great responses!
I’m moving towards it not being quite worth the hazzle of setting up passthrough, for me, even though I like the idea of it in theory. Like @wertigon says I might be fine for a forseeable future with just Steam Play in Linux. I’d rather have the full power of the system accessible for computational tasks, cuda enabled etc in my main OS. Maybe put a Win10 installation on a separate drive instead that I could boot into for the occasional game not working good enough in Linux.

8

That works, of course, as long as you don’t mind letting Windows 10 run un-sandboxed. I do mind, which is why I’m willing to put up with the gymnastics.