"Sharing" a GPU between virtual machines?

Hardware Hub GPU
1

Hey levelonetechs!

I’m interested in making the transition to Linux (from Windows), I’ve been interested for a while now, but last time I switched I just ended up running Antergos and doing all of my computer usage inside of a Windows 10 gaming virtual machine… It really defeated the purpose of having my host being in Linux. (Besides the learning I did!)

It seems that enthusiasts and end-users have been making great strides to close the gap between corporate computer/server uses and with the competition from Ryzen, hardware manufacturers are finally starting to trickle some bits of enterprise tech features (i.e. many processing cores) down to consumers!

A while back, in the LevelOneTechs video on Qubes OS, I remember hearing that GPU sharing “wasn’t quite there” primarily because of a feature missing (or not being enabled) on most consumer-grade GPUs. I might be mis-remembering what video I heard Wendell talk about this, but this segues into the use case I’m trying to ask about:

I’d really like to build a system that is capable of sharing my video card between my host OS–probably Qubes OS–AND whatever virtual machines I’m running. Simultaneously. I’m familiar with GPU passthrough, but that’s not what I’m looking for, I want my Host OS to continue to be able to use the video card and display my desktop… if it would be possible to run my gaming VM in a window inside of my Linux desktop, that would be amazing.

The main complication with other solutions is managing my monitors and connecting and driving them through different video cards/sources, whether I’m using the iGPU/APU form my processor to drive my Linux Desktop and giving my video card to a VM or using multiple dedicated video cards or whatever else… I’m looking for the elegance of just running everything under one video card in a more ‘seamless’ and ‘flexible’ environment. And I understand that it might not be out there.

From what I understand, higher-end/‘professional’ Nvidia cards (Quadro or Grid? I’m not sure exactly which) are designed to be put in servers and more or less shared between virtual machines/clients, and they have some sort of feature enabled (or “physically present?”) that allows them to be used in that manner. Are those video cards capable of being used in the manner I’m referring to? Are there any cards that are more consumer-accessible/less expensive? Realistically, I could budget for any video card under ~$1,000, but unless I’m getting something powerful enough at that price point to drive 4K at ~120fps, (which, to my knowledge, there aren’t really cards that fit that bill) it doesn’t seem worth paying that premium when that’s where gaming is headed in these next few generations.

So, I guess to make my questions more succinct:

  • Is it possible to “share” a video card between a host and its virtual machines like I am describing?
  • Are there any video cards currently capable of doing that?
  • Are there any consumer-accessible (by your own metric) video cards capable of doing that?
1 Like
2

The feature you’re looking for is called Single Root I/O Virtualization (SR-IOV). Or at least, that is the name of the version standardized by PCI-SIG.

Allegedly, Intel Iris Pro integrated GPUs support SR-IOV (branded as “GVT-d”). This would really be the only affordable option at your given price limit. However, I’m not sure if the work has been done in Linux or libvirt to expose the feature. Linux does support SR-IOV for many server network adapters.

SR-IOV is also supported by at least the following AMD cards (branded “MxGPU”)):

  • Radeon Instinct MI25
  • Radeon Instinct MI8
  • Radeon Instinct MI6
  • FirePro S7150
  • FirePro S7150 X2

It’s difficult to be sure that this is an exhaustive list. Nvidia has cards which offer similar features, which at least used to be branded “GRID,” but I think the older Kepler-based ones used a proprietary technology instead of SR-IOV, and I cannot confirm either way for the newer ones. Here are lists anyway:
https://en.wikipedia.org/wiki/List_of_Nvidia_graphics_processing_units#Grid_GPUs
https://www.nvidia.com/object/nvidia-grid-buy.html

Some of these cards can be found on Ebay for less than 2000 USD. They all have passive cooling designs which require a server chassis or similar environment. Additionally, I don’t think any of them have display adapters so you will still need integrated graphics or a separate card to attach your monitors. These cards are meant to be used with enterprise hypervisors (VMware, Citrix, etc.), so I’m not sure if the work has been done in open source code to enable them in Linux. Also, the intent in this case is to share the GPU between multiple guests, not between the host and a guest. I’m not sure if that makes a difference in implementation.

You will still need a remote console viewer to see graphical output from your VMs. You can achieve a similar experience for 1 VM per spare discrete GPU by using GPU passthrough with an ordinary GPU and streaming the desktop instead of connecting monitors.

1 Like
3

So, the ideal use-case would be that I have ~4 monitors hooked up to some sort of display adapter, the one driving my host machine.

The idea being that I can open up a ‘window’ of a virtual machine and have it fill up one of the monitors. Hopefully, it would be possible to open up a second window and fill in a second monitor and have that operate as a dual-monitor Windows gaming VM. While still having that ‘meta’ access to the host and the other two monitors. (And theoretically ‘tab out of’ or minimize the gaming VM if I felt the need.)

In theory, I could hook up to a processor with integrated graphics, but those are typically limited on video-outs. Or I could hook up to a low-end ‘enterprise’ GPU with enough video-outs, kind of wasting a PCI-e slot and bandwidth. The low-end gpu solution is attractive, because it allows me to not be prohibited from choosing a powerful platform for virtualization that doesn’t have integrated graphics like a Ryzen 7, but the extra clutter of the extra GPU puts me off.

But, I still would need a video card to dedicate to my virtual machines. I’m of the understanding that if I pass a video card through to a gaming VM, it’s blocked off from being used by pretty much anything else. So, I’m effectively again limited to running a Linux desktop with only one gaming VM open. Can I have other virtual desktops running, whether they’re just running browsers or otherwise doing things that need to output video?

4

For a gaming VM, you want GPU passthru. That’s all there is to it.

If you only want “lots of monitors” then your host OS wouldn’t need a particularly high-end GPU. Just something that supports the resolution + number of monitors you want. For example, I used to run a 1440p ultrawide and two 1080p monitors on a 750 ti — which is something you can pick up secondhand for around $40.

In the end, though, a second GPU isn’t “clutter.” Not if you’re actually making use of it.

Assuming your “other virtual machines” didn’t need a GPU, and that you had sufficient cpu/ram to dedicate, then sure. no problem.

5

What I’m trying to do–in my hypothetical situation–is be able to run a gaming VM on my host machine with the “lots of monitors”, and be able to kind of dynamically allocate my system’s monitors to the gaming VM and its GPU.

With GPU passthrough, I’m under the impression this would require connecting my host machine’s GPU and the GPU being passed through to the monitors. I’ve seen VMs scripts disable a particular monitor on the host and let the monitor auto-switch over to the now-active GPU, but this isn’t as fluid of a solution as I was hoping for.

I was wanting the VM to run in a ‘window’ on the host OS that I could move to whichever monitor I wanted, then make it fullscreen in there. And this hypothetical solution would also allow for dynamically creating a second ‘mintor/window’ for the VM that I could fill up a second monitor on the host with. I don’t believe GPU passthrough works in this situation, but maybe it would? I don’t even know if what I’m talking about is actually doable, but it’s the pipe dream.

6

I the cheapest/simplest way to meet your requirement would be a physical windows box that you stream games from.

The SR-IOV or Nvidia grid cards would meet it, but at what cost, and performance level?

Maybe one day…

Edit: if you have not read it yet, this thread might help, where we kicked around the same subject -

7

Apparently AMD has SR-IOV support in their linux drivers now:
https://www.phoronix.com/scan.php?page=article&item=amdvlk-radeon-vulkan&num=1
The end of page 1.

Still trying to work out what actual GPUs support this though…

3 Likes
8

so then… how do gpu passthru?

9

If I were building a rig like this there are a couple routs I would look at. You could use GPU pass through with a capture device on the host OS. You can then display it on any of the host monitors in a window without the encoding overhead. A second option is to take all the monitors and put them through a matrix switch to switch between host and VM GFX outputs and route displays around. You still would need either a separate mouse and keyboard, or some mouse/keyboard sharing software to be able to navigate between VM and host.

The problem is many cheap HDMI matrix switches are very slow to achieve signal lock due to HDCP implementations. Proper broadcast grade gear like I am used to working with would rival the cost of most gaming computers, but could be done for a few $$$$. Those options still have frame rate limitations to 60fps though which could be an issue if you like high frame rate gaming.

I will be looking around for a GPU pass through instructional also, but there are a few things ahead of that on my todo list.

Edit: instead of using a capture device or matrix switch, take a look at Looking Glass. Not sure if they support multiple monitors yet, but it would be the way to go.