A few questions for before I take a deep dive on the subject

Software & Operating Systems VFIO
1

It’s been 8 years since my last GPU upgrade and with the lockdown in effect, I think now is a good time to finally upgrade. I’d like to try VFIO / Looking Glass but I have a few questions that I’d like answered before I start going in depth. Just to save me wasting my time.

Could someone please help a noob out and answer the following questions?

  1. Gaming on both the Windows Guest and Linux Host.

I’m trying to game mainly on Linux but I still have to hold my nose (so to speak) and use Windows for certain games or applications. But I don’t want to be unable to use the new GPU on the Linux host.

Can I easily and quickly switch the passed through GPU between guest and host, even if that means rebooting, without having to do too much work?

  1. Could I switch the passed through GPU between different guests?.. i.e. shotdown the Windows guest and fire up a Linux guest with the GPU passed through?

  2. Red vs. Evil… I mean Green

Unfortunately one of my key areas of interest is silent computing, which means low TDPs and power draws are big pluses. As such, I’m pretty set on an MSI 1660 Super Gaming X as it has a low TDP and appears to be one of, if not the quietest 1660 Supers.

My question is, is there anything that AMD has to offer, when it comes to PCI passthrough, that’d make the compromise on thermal and acoustic performance, and power draw worth it?

  1. A second GPU for the Host.

I’ve currently got an Asus GTX 670 DirectCU II TOP that I plan to use for the host until I know whether replacing the host GPU is worth stretching my budget.

My question is, will the Host and Guest both using Nvidia cards cause any issues?

  1. Furthermore, if I do upgrade the Host GPU, it’d likely be a low end Nvidia GPU for the same reasons described above. Is there anything I need to be aware of other than not using identical cards?
2

It’s doable, really depends on which GPU you get, but it’s not ideal.

It’s easiest with Nvidia GPUs because their drivers (despite being closed source) are built more or less properly. (fully featured)

Yes, absolutely. This is as easy as shutting down one guest, and starting up the other.

For the “host gpu” (the one you use on the Linux os on the Metal) you can go with whatever, but Nvidia is going to be the best option for actually passing it through.

So, AMD cards are great if you undervolt them a bit. AMD should have really done that from the factory, but they didn’t.

But, again, AMD isn’t the brand you want to go with for passing through. Their compatibility is hit or miss, and Nvidia is as close to a sure thing as you can get.

Nope, I’m using a 1070ti on my host and a 1080ti on my guest. (typing from the host right now) and it works just fine.

You actually can use identical cards, but it’s not quite as easy.

But, there are some super-low-end gpus that just don’t have the power to do what you’re trying to do, so my recommendation is to not go any lower than a 1050ti class card.

Even though Looking Glass does the rendering of the game to your VM and the passed-through GPU, you still need a little bit of horsepower to render the looking glass client on your Linux machine. While it’s doable, even with an iGPU, your best experience will come from a slightly more beefy GPU.

So given the questions you asked already, it seems like there’s a few other gotchas that you might bump into, so I’ll try to educate you about them ahead of time.

What sort of system specs do you currently have for your other components? Certain motherboards are more compatible with passthrough than others. Certain CPUs obviously work better than others. I typically recommend at least 16GB of ram to get this done.

If you’ve got more questions, I can definitely help you, feel free to ask.

3

It’s doable, really depends on which GPU you get, but it’s not ideal.

It’s easiest with Nvidia GPUs because their drivers (despite being closed source) are built more or less properly. (fully featured)

…

Could I switch the passed through GPU between different guests?.. i.e. shotdown the Windows guest and fire up a Linux guest with the GPU passed through?

Yes, absolutely. This is as easy as shutting down one guest, and starting up the other.

As long as I can switch the GPU between guests, host access isn’t too much of a big deal. I do have a follow up question regarding switching the guest GPU back to the host but I’ll save that for a later post if I may.

Red vs. Evil… I mean Green

For the “host gpu” (the one you use on the Linux os on the Metal) you can go with whatever, but Nvidia is going to be the best option for actually passing it through.

That’s helpful to knnow because I was under the impression that AMD was easier but Nvidia had better performance.

So, AMD cards are great if you undervolt them a bit. AMD should have really done that from the factory, but they didn’t.

But, again, AMD isn’t the brand you want to go with for passing through. Their compatibility is hit or miss, and Nvidia is as close to a sure thing as you can get.

As much as I’d like a fully FOSS system, sometimes it’s just not pratical or worth the comprimise… why do I feel like I’m slipping down a slope when I say that?

Nope, I’m using a 1070ti on my host and a 1080ti on my guest. (typing from the host right now) and it works just fine.

Thanks.

Furthermore, if I do upgrade the Host GPU, it’d likely be a low end Nvidia GPU for the same reasons described above. Is there anything I need to be aware of other than not using identical cards?

You actually can use identical cards, but it’s not quite as easy.

But, there are some super-low-end gpus that just don’t have the power to do what you’re trying to do, so my recommendation is to not go any lower than a 1050ti class card.

Even though Looking Glass does the rendering of the game to your VM and the passed-through GPU, you still need a little bit of horsepower to render the looking glass client on your Linux machine. While it’s doable, even with an iGPU, your best experience will come from a slightly more beefy GPU.

That’s a very good point, I hadn’t considered the host GPU doing much other than high bitrate video playback. I had initially considered a GT 1030 but I’ve pretty much ruled it out as I’d like 3 video outputs, which means more the GTX 1050 range.

So given the questions you asked already, it seems like there’s a few other gotchas that you might bump into, so I’ll try to educate you about them ahead of time.

I will. The point of this thread was basically to confirm whether or not I’d be making a mistake going the Nvidia route and whether I could pass the GPU from one guest to another. Any other gotchas, as long as they’re not total deal breakers or they stop me from doing what I described in my previous post, then I don’t mind working around them.

What sort of system specs do you currently have for your other components? Certain motherboards are more compatible with passthrough than others. Certain CPUs obviously work better than others. I typically recommend at least 16GB of ram to get this done.

My current specs:

Ryzen 7 1700x
Gigabyte AX370 Gaming 5*
32 GB of RAM
500GB NVMe disk for Debian (Root on ZFS)
+a variety of SSDs at my disposal

*I did have it’s IOMMU groups somewhere but I can’t find it at the moment. I’m just about to eat dinner so I’ll get a copy of them and report back tomorrow if that’s okay.

If you’ve got more questions, I can definitely help you, feel free to ask.

Thank you. You’ve been very helpful so far but I’d be happy to pick your brain in the next few days.

One question that springs to mind, and I’m guessing you won’t be able to answer with any degree of certainty until I get the IOMMU groups output, is how I’ll address the question of audio. I use an Asus Xonar DGX sound card but if at all possible, my motherboard has two audio chips, on for the front i/o and one for rear i/o. I’d like to be able to pass just one of them through to the guest. Have you ever heard of that being done?.. is it just a matter of them being in differnt IOMMU groups or seperating them with the ACS(?) patch?

(worst case scenario, I pass the sound card through or buy a USB DAC)

I have got more questions but dinner is on the table.

Thanks again sgtawsomesauce!

4

The complete answer to this is that AMD is easier when it works, but it varies, not just between GPUs, but even between AIB partners. So, for example, people reported that the R9 fury didn’t work. My Sapphire Fury Nitro worked. People reported the RX580 did work. My MSI Gaming X 8GB 580 didn’t.

Now, Nvidia requires tricking the drivers to think it’s not in a VM, which reduces performance by 0.5-2%, and is also very easy (just a couple config options in the xml) But I’ve never encountered a card that I couldn’t pass through. 660 ti, 960, 970, 1060, 1070, 1070 ti, 1080 ti, are all personally confirmed by me.

That’s why I recommend Nvidia. Because with AMD, even if someone say “5700xt works” that might mean that the PowerColor one works, but ASUS does not, or the reference model one from ASUS works, but the overclocked one does not. (for example. I don’t know for sure)

Now, I’m not 100% sure how much exactly, you’ll need, but if you are gonna buy a GPU, the 1050 range is a much better value proposition anyways.

Looks good. I don’t know personally about the AX370 Gaming 5 board, but I do know that X370 chipsets have fairly good support. (generally)

If you want first-class support, you may end up needing the ACS patch, but it’s not too difficult to make work and honestly, I use that patch, even on my 1950x system.

So, onboard audio tends to be a no-go, but it’s possible. What I do on my system is get a PCIe USB 3.0 controller, pass that through and plug in a USB audio dongle. They can be had for pretty cheap. I’ve got a number of them, and it works pretty flawlessly.

So the ACS patch overrides the security settings of the motherboard. It doesn’t actually change things, it just sort of tells the kernel “go ahead and break the rules, consequences be damned”

I have (what I consider to be) a good explainer on this that I wrote a few years back, still relevant, linked here:

This guide also runs you through how to do the passthrough process, but it’s a bit outdated.

5

Hey sgtawesomesauce,

Appologies for the slow reply.

There was an 11th hour alternative the the 1660 Super (just after I’d actually ordered it), so I’ve been doing some research into a new option for the GPU. I ended up ordering the 1660 Super, which arrived two days early. So I’ve been kind of sidetracked.

I was all ready to respond to your previous comment when the 1660 Super decided that the nice warm bed of my PC would be better if it shat in it. I’m pretty sure the 1660 Super has going the way of the dodo / a certain Norwegian Parrot, and is either dead or dying.

So if you don’t mind, I’ll get back to you in a tomorrow. Once the RMA is sorted, and I have had time to get the IOMMU group output and do some research into decent USB cards.

1 Like
6

Excellent. Take your time. I’m here freqently.

And have fun with the RMA process.

1 Like