GPU Passthrough with KVM: Have Your Cake and Eat it Too

Software & Operating Systems Linux
, ,
161

Nope never messed with the mouse and keyboard so they are how ever the default configuration for the VM is, I did pass through the Logitech G-13 game pad, loaded the software/drivers in win7 and it works as expected, and I'm still using the bridged NIC because I need to make up a cable to connect the Intel NIC I passed through, the host is DL'ing some stuff while I was playing which I'm sure was contributing to some of the lag in the video, funny the audio in GW2 doesn't seem to be laggy like it was in Borderlands 2..

@lessershoe, I do want to personally thank you for this thread, it inspired me to keep trying to get this to work and so far I'm totally impressed....thanks!

1 Like
162

A fitting post for Independence Day......

Adobe Illustrator CS6 running on Win7 In a KVM...

Adobe Dream Weaver CS6 doing the same...

And Adobe Photoshop CS6 running on Win7 in a KVM.

I didn't try Premier, didn't even install it, it might work I'm not really sure since every time I've used it it was a huge resource hog and I don't really do videos any more.

Happy 4th of july guys!

163

This would be the ultimate way for Logan to render The Tek! No more Premiere crashes. Also lets him run all his Adobe programs natively too.

2 Likes
164

Kind of a follow up on GW2 just in-case someone might be interested....

Background: So I had a interest in running a KVM pretty well with the sole intent of running games, especially GW2 since it's one of the few games I play, I wanted to move to Linux but knew I had to either dual boot or run a VM to keep GW2 since I doubt it'll ever be ported over to Linux. Running Adobe (still need to try Corel Draw) was a added kinda' proof of concept for me to first show it would work and second to maybe in the future switch my wife (graphics designer) over to Linux.

Current: So I'm on my seventh iteration of building a VM using KVM on Fedora 22, as you might be able to figure out I've gotten pretty comfortable doing so (and may write a guide to help others if there's any interest), after all my testing I deleted my last VM (#6) yesterday to build what I hope will be the last or final (no longer testing), I had originally given the last several VMs a entire 500g HD which was fine for testing but if your going to run Windows and have games & applications it isn't enough for a permanent install. I had also in every testing VM given Windows about half of the system resources that were available (ie 4 of 8 cores, 16g of 32g ram, bridged NIC, and one 270x).

So as I said finished testing and feel comfortable enough with what I'm doing to think I can now build a VM that I can use and build on, the current VM I've built will be a bit different I removed the 500g drive and replaced it with a 2tb hard drive, upped the memory from 16g to 24g and from 4 cores to 6 cores of the CPU that should leave Linux with more than enough to operate and not suffer to much, I also gave the guest it's own gigabit NIC which seemed to improve some of the latency I was having with a bridged NIC while working in the host and guest at the same time. My eventual plan is to swap the video cards from host to guest now that I have a handle on the PCI pass through and have successfully accomplished it numerous times.

My current daily driver has 3 R9 270x cards in it, two are being used by the host and one by the guest, I'm going to swap that giving two in crossfire to the guest hoping to improve the video performance in windows games, and Linux should easily be happy with one card driving 3 monitors at a 1080p resolution (1920x1080).

To get GW2 to run acceptably with one 270x in the KVM I had to turn down all the settings to medium or low, it's fine that way but we all know we want as much video that we can muster for rendering and while playing a MMO there are times when lag becomes a concern, I want to avoid as much of that as I can and I hope passing both cards through will give it the needed resources to work as it did running Windows on bare metal.

So where I'm at right now is the KVM is built and configured as I stated above with the exception of the passing through the two 270x cards, I'm going to run it like this for a week to get a feel for how it performs then this weekend swap the cards and give that a try, hopefully I'll see a difference.

I do still have a mouse/polling issue in Fallout 3 that no matter what I do I can't seem to correct, it's there regardless if I use the default settings for the KVM or if I pass the mouse through, I'm not really sure exactly how to resolve that issue since it's the only game that exhibits the behavior and it makes the game pretty much unplayable, I'd really like to correct this before Fallout 4 come out just in case it's specific to the engine/franchise.

Another thing I'm seeing is the KVM using multi cores but one core gets slammed to 100% quite a lot and it's not the same core it moves around from core to core, not really sure if that's normal or not, if it's a hardware bottle neck of some sorts, it doesn't seem to create any lag that is detectable and when I shut down a game it returns to what I would call normal spanning cores at 0-50% load but moving from core to core while the Linux CPU cores remain at 0-20% most of the time (unless it's installing or some other task), I made a video the other day of it and when I get a chance I'll post it so you guys can see what I'm talking about.

Anyone have any tips, tricks, or ideas?

Sorry for the rambling and lack of a tl,dr..... :)

165

With core usage you can pin the process to what ever cores you need.

https://www-01.ibm.com/support/knowledgecenter/#!/linuxonibm/liaat/liaattunpinproctop.htm

https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/5/html/Virtualization/ch33s08.html

1 Like
166

If you were up for more trial and error and potentially reinstalling Window a few more times, there are a number of things you can try to improve guest performance. They can all be found here:

http://vfio.blogspot.com/
https://bbs.archlinux.org/viewtopic.php?id=162768

I think it would be beneficial to read through them but there are over 200 pages, so I'll list things that I recall:

  • You are using an AMD graphics card so you can enable hyperv flags. The options I made a post about:
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>

and

<timer name='hypervclock' present='yes'/>

can all be added to the config to improve Windows guest performance

  • The config options I posted regarding CPU pinning generally improves guest performance. This may improve your multi-core load handling too.

There are also:

  • Hugepages memory backing
  • Virtio devices

I'll see if I can expand on them when I get around to it.

1 Like
167

LOL...I'll never be totally finished testing, I know going forward there will be advancements in KVMs, Linux, and just about every aspect of virtualization as it pertains to what we (I) am trying to accomplish, it's the question "is it good enough?" in my case I think there is always room for improvement, always more tweaks to enhance the performance and stability of the guest.

I guess I should've just said resource/hardware-wise (with what I have to work with) I have finished with that aspect of testing, the current configuration of guest resources seems to work good enough for what I'm doing, but the goal is stability and performance that is the same or better than bare metal.

The blog as been my bible so to speak, without Alex's blog I doubt I would have been able to accomplish the pass through so quickly.

I've looked at the archlinux site you linked a couple times to find answers, I do need to start at the beginning and read the whole thing, I know I've barely scratched the surface in what I've learned so far.

I was just reading about this yesterday along with paravirtual drivers.

Thank you Sir.

On a semi-related note I ran furmark on my guest last night to test the stability, topped out at 70fps but what kinda' has me worried is the gpu temp shot up to 98c after about 15 minuets, it did level out there and stayed between 96-98 degrees celsius, took it about 7 min to reach that temp then pretty well flat-lined at 97 degrees, I did a screen grab at the end I can post if anyone wants to see it. As you can figure I'm trying to get a baseline performance reference for when I swap the cards being passed through....Oh and the GPU fan was at 100%.

168

Has anyone had any luck with DirectX 11 games using this method? This is pretty much the biggest thing keeping me from hard-switching to Linux.

169

It works.

1 Like
170

You pretty much give Windows an entire graphics card to its self so any thing Windows can to the KVM VM can do

1 Like
171

I solved the polling issue in Fallout 3 last night.......lol

Plugged in a newer mouse to one of the USB 3.0 ports, passed it through to the guest, fired up Fallout 3 and the mouse works just like it's suppose to, sure makes a difference when QEMU/KVM recognizes a device for what it truly is, and windows has built in drivers for it! ( I guess using my old Logitech TrackMan Marble FX wasn't the best idea....I feel so stupid. )

172

How are you passing in the USB devices? You can pass them in individually as USB or you can pass it the whole USB controller much like you would other hardware. I prefer passing in the controller myself but I would say there are pros and cons to both methods.

Regarding

You may have already seen it but both are covered very well in this post:

Hugepages may be a bit tricker. I believe regenerating initramfs as discussed in the blog is optional. You can create a file:

/etc/sysctl.d/40-hugepage.conf

and populate it with

vm.nr_hugepages = 4096

Restart the host or run the following to make change at runtime:

sysctl vm.nr_hugepages=4096

Add hugepages memory backing entry to VM config:

<memoryBacking>
  <hugepages/>
</memoryBacking>
1 Like
173

I pass the keyboard, mouse, and a game pad through as individual USB devices, I thought about just passing the entire controller through but decided against it because the host is also using devices on the same controller, I believe the MB has two USB controllers one 2.0 and one 3.0 but to keep things flexible and maintain connectivity for both the host and guest with external devices it just seemed to make the most sense to only pass the devices through that the guest truly needed.

One odd occurrence I've noticed is if I stick a USB thumb drive in a 2.0 port (case has 2.0 ports on the front and 3.0 on the top) the host sees it but the guest doesn't, if I stick the drive in a 3.0 port the guest and host both see the device, I'm assuming that has something to do with USB 3.0 connectivity and not a issue with QEMU or the guest as I have it configured, I'm assuming that QEMU/KVM by default passes some devices through using virtual drivers if it see's the system has that device available and has the needed support to do so.

I really need to work on the CPU core utilization, it's a issue that I can see happening, the guest (win7) sees the CPU as having only one core which makes every program running on top of it also see only one CPU core, GW2 when installed gives me a warning at startup that there is a hardware error (deficiency) the error states that the CPU only has one core and isn't optimal to play the game. Also running CPU/GPU benchmarks programs they report back that the GPU is more than capable but that the CPU is only one core therefor reducing the system performance.

During the QEMU/KVM configuration in virt-manager when I look at the default it wants to set the CPU up as 6 sockets with one core per socket, I have been changing this every time to one socket with 6 cores and 6 threads, it tells me the CPU doesn't support hyper threading (FX does not support it, so that is correct), KVM/QEMU wants to use a Opteron XXX CPU and the actual FX CPU isn't listed in the drop down menus, I have been just leaving the CPU at the Opteron it is choosing thinking it is smarter than I am at this point and is doing some form of auto-detection?

But under the hood so to speak I can see multi-core utilization going on by running a monitoring program in the host, all of the 6 cores being passed to the guest get utilized in like I said the other day a spanning process where the CPU load looks like it's moving from core to core, all the games I've installed work better than I ever expected them to run in a KVM with the exception of the occasional audio latency that @lessershoe also has in his setup, the audio is something I can live with or if it becomes a big issue in the future I can install a USB sound card to hopefully solve the problem, I really have no other latency issues with anything else be it hardware or software running in the VM since I gave the guest it's own NIC.

The interesting thing to me is that the host and guest act like (and really are) two totally separate machines that are connected together like Siamese twins...lol it's a very interesting and highly flexible configuration running Windows on top of Linux in a KVM so close to giving you the feeling that you have two bare metal installs but sharing the same peripherals and so far has been extremely stable, and the fact that I don't run any windows antivirus or firewall software removes a lot of overhead which I believe adds to the stability of the guest.

When I started down this road I had the misconception that running a KVM would diminish the host in some way, be it stability or resources, useability, coming from a lifetime of using windows you get accustom to the constant drain of system resources that some programs cause and the lack of stability that eventually cause a crash or a necessary reboot over time, Linux has none of these issues, doesn't seem to care about the KVM it's just another process to run.

Thanks again for the help and advice.

174

Rock on Man.

Until my fury show up I will be setting up a 750ti for my host and a 280 for the VM

When the fury come it will be for the host (Suse 13.2) and the 280 will stay with the VMs (Suse Tumbleweed and Windows 10 tech preview) Normal The release version of windows will be dual booted with my Linux/KVM setup. Sometimes I just want to play a AAA game. Linux setup will be handy for much of my testing.

1 Like
175

@lessershoe, this is such a sweet setup, I played Fallout 3 last night for a few hours and I really couldn't tell it wasn't on a windows bare metal install, it was pretty well flawless except for the occasional audio glitch, I'm really interested in how the Fury/Fiji works out it's not in my comfort zone price-wise but in a year or so it might be a good upgrade option for me to consider.

176

It would be awesome if you could test passing through the Fury and playing around with it even if you don't intend to keep the setup. I would really like to know how or if it works. I'll eventually get a Fury but likely not until the Nano.

177

if its an AMD card then it will likely work just fine

178

I will see it it works for science

3 Likes
179

Thank you

Unfortunately this hasn't been true for some of their chips like Hawaii where the device cannot be reset and reused after restarting the VM unless the host is also restarted. As far as I know this was mostly but not entirely addressed in Qemu 2.3. I have a R9 285 that still has this behavior and doesn't work.

180

That's why I said likely will work. Nothing is ever guaranteed. Especially things this insane.