Upgrade for GPU Passthrough: Is it feasible?

Hardware Build a PC
1

I'm planning an upgrade and am tempted to give running windows in a VM with a GPU passed through a go. However, I need some opinions on this route, as there's a significant difference in the two options (passthrough or ...pass) and I'm not sure which is best. I also need to understand VFIO / how VM's handle things better.

So, budget: ~1000 AUD, current parts (* marks parts I intend to re-use). Purpose: 3d modeling (polygonal, not cad, don't suggest workstation GPUs — mainly I need ram and vram, then CPU speed, then GPU speed).

i5-2500k
* CM H212+ cpu cooler (Need to check what sockets it supports)
16gb ddr3 ram
* R9 380 4gb (new-ish)
* Corsair RM 650x (new)
* 120gb SSD + 2 mech hdd's
Old NZXT Case

* Peripherals:

23" IPS
Headphones (3.5mm)
KB / Mouse / Intuos tablet

Plan A: GPU Passthrough

$ 269 - i5-6500
$ 155 - GA H170-HD3
$  95 - CM Master Box case
$ 260 - 32 gb ddr4 ram (I need more than 16 regardless of passthrough or no)
-----------
= 779

This leaves the possibility of either a1) getting a 1050 ti** for the linux host instead of using the cpu's video output ($245) or a2) getting a second monitor. If I do that I could get the 1050 later down the line. I still need to look into VFIO more to find out how much of a hassle it would be to use one screen, plus a 2nd screen would make normal work smoother (less tabbing). I'm also not sure yet how audio would work between the host/guest with 3.5mm headphones, as well as best options for mouse/kb sharing.

** why 1050 ti? Several reasons, 1 in AU stores it's significant cheaper than 470 (250 vs 340), and 2, lower TDP (75 vs 110). I'm kicking myself for not spending the extra 20 on the corsair 750w, but so be it. And 3, some software I use plays better with nvidia, some with amd. Some software works on both windows and linux. So the flexibility of having each brand works in my favour. Anyway I don't need great performance on both host and guest, but extra vram is welcome.

Also I just noticed (option a3) that a store has the following deal: $600 for xeon 1230 E5, 16gb ram, mobo. + extra 129 for another 16gb for total $729, cheaper than i5 6500 build, less CPU options but better CPU to begin with (threads + higher clock). Does it make sense to consider Xeon over i5 for passthrough, or otherwise? Are there any non-obvious differences I need to consider? Higher TDP is one (80 vs i5's 65). Also the iGPU is disabled on the Xeon so this forces buying the 1050 with Plan A.

Plan B is very similar to plan A hardware wise, excluding the extra/new GPU. I see plan B like this:

+ Keep It Simple, Stupid
+ Cheaper, not potentially wasting performance
- Windows...

Other thoughts:

How much performance (CPU wise) would I lose with passthrough and half stuff running on a VM? What if it can't be halved like that, and 90% runs in the windows VM (like without getting the 1050?)

$129 for intel 600p m.2 ssd... (An indulgence, not worth it)

Why can't I re-use the case? I could, but my current r9 380 JUST fits in, and it's a massive pain to slot it in. Plus it's pretty rusted at fan ports etc, fair bit of dust gets in, very messy cables etc.

Could I just use linux natively and forget windows / dual boot? This might be a legitimate option, as all my vital software can work natively, I'd mostly lose out on minor things + new things that come out lacking linux support. The main problem here is GPU drivers in linux, with CCC I get bad perf in one program, with open source drivers a different program doesn't work. Both programs are vital. At least with passthrough I'd have flexibility to work around that stuff. Maybe I could sell r9 380 at a loss and get a nvidia gpu and hope like hell it plays nice with everything on linux... I'd still have to boot into windows for the odd game though.

Would it be viable to keep my 2500k, overclock like a boss, get more ram? Or would putting extra money into this be a lost cause? It's quite old and I'd rather not be screwed when something else dies.

Anyway, thanks for looking, hit me with any suggestions. I think I mainly need to figure out if a passthrough setup is worth the hassle in my situation.

2

I would go with that server option.

But I would use the extra money to get an extra monitor. I would also get an extra keyboard and mouse.

Then you could just pass through all the things. It would be like having a windows and linux machine running side by side.

3

The i5-2500 non-K has VT-D and all that goodness, so you could try to score one of those second-hand. Plenty of people will be willing to do a straight swap or even give you some extra money for the K.
The multiplier on the non-K version only goes to 41 (giving a theoretical maximum clock speed of 4.1GHz), so no need to worry about buying one that has been degraded because it ran at its limit for years.
You might even be able to get the i7-2600 non-K, which would be even better due to hyperthreading.

Dual boot is also an option. If you get a case that has a hotswap bay built in (or has a hotswap module in a 5.25" bay), you can set the hotswap as first boot device, then your linux SSD as second.
On my main PC I run it like that. Runs Linux most of the time, but whenever I put the Windows SSD in the hotswap bay it boots from that.

4

As someone that has been running a pass through system for about 18 months I applaud you for thinking and planning, hardware pass through isn't something you just do it is something you plan out and buy the appropriate hardware to build a system that is robust and stable.

The proper way to plan this out is to look at it as two complete systems in one case, you must have enough resources to share so that both the host and guest work as you want, this means lots of cpu cores, lots of memory, enough drive space for two complete systems, plenty of buss options (usb, sata, etc), then of course input/and output devices (keyboard, mouse, monitors)

Best advice I can give you is not to skimp on the CPU, you need cores the more the better, look at the spec of the OS you want to run as a guest (not the minimum) and the requirements of the software you want to run in that guest environment, clock speed is way less important then the number of cores you have to share, look at what you plan to share with the guest and ask yourself would you build a stand-alone computer with those specs and what level of performance would you expect? use that as the hardware spec you want to achieve as half of what you will need to run both systems in one box (case).

There is a performance hit for running a KVM on the guest side it isn't 10% but is probably something close to 5% of you resources, if you plan for it it will never be a issue.

A couple other things to consider is the host system, not every Linux distro is suited to run a KVM with hardware passthrough, IMHO distros based on Debian like Ubuntu while will work are not the best choice, OpenSuse, Fedora, Arch are great choices because the level of support built in the kernel is more up to date and complete, you will need a understanding of the CLI and basic Linux commands, and not be scared editing files, there is a certain amount of trial and error in doing this but the rewards are well worth the effort if you follow a plan and give the machine a fighting chance with enough hardware to do what you ask of it.

There is no shame in dual booting...none, but it has no "cool" factor, if dual booting will satisfy your wants and needs then that is the way to go it's much less hassle and will get the job done.

If I can help you in any way just ask and I'll try to steer you in the right direction.

2 Likes
5

Thanks for the advice.

I'm pretty set now on the Xeon build. $983 gets me the xeon E3 1230 v5, 32gb ram (2133hmz dual channel ddr4, which is one downside to the xeon, but I need quantity more than speed anyway), MSI Xeon E3 Krait Gaming V5 mobo (which I cannot find a single review of, but at least the chipset seems to support VT-d, will need to find out something about bios), and the 1050ti. Which is much better performance CPU-wise than going for the i5-6500 route, and cheaper. Luckily I fit into the weird category where the xeon makes sense.

The only big thing I miss from that build is the case, which is +99 extra. Should be doable but I'll think about it. I also just checked and apparently my old CM hyper 212+ does support 1151 because the holes are the same between it and 1155/1150, which is good news.

I dunno, I think that would be very awkward to work with. Having to change kb/m every time I want to use an application in the 'other' machine. Hard pass on that one. A software solution like synergy seems better. But I still need to research passthrough more and figure out what options are available re devices and audio.

6

I can vouch for synergy, once you get it set up it works flawlessly; the nightly builds also don't require a license. Qemu allows for passing through guest audio to host (requires some config changes, see passthrough guides). As for guest audio, the ac97 soundcard works great, you just need to disable driver verification on windows 8 and up (due to the latest drivers being for win7)

7

You will need duplicates of a few pieces of hardware like NIC (you will want and need two discrete cards one for host and one for the guest, this is especially true if you will play any games on your guest) you will find that you will need a solution for sound latency (it's is going to happen) doing productive stuff it will not be a issue, but again if you plan on gaming on the guest you will want a USB sound card, keyboard and mouse are easy, you can use Synergy, a KVM switch, or just use separate input devices. I use a separate mouse (trackball) for the guest and share a keyboard.

So to do this you will want to also passthrough USB ports or a entire USB controller (best option IMHO) you will find that most virtual passed through devices work reasonably well like a DVD-ROM or other optical drives, most anything USB that you can use through a passed through USB controller will be recognized in the guest system and work as expected.... ie card readers, keyboards, game pads, mice, etc.

So something you should think about and consider is that the Xeon is a 4 core CPU, the best you can hope for is to pass 3 cores to your guest leaving one core to run your host, ideally you would want to just balance that and pass 2 cores and leave 2 cores, so the thing to consider is would you run the OS and software that you plan to use on a dual core machine? in my mind a Xeon E5-2670 which is 8 cores would be a better choice because it would allow you to give 4 or 6 cores to the guest while leaving 2 or 4 cores for the underlying host.

A lot of people think of pass through KVMs as like dual booting where they will only use the KVM when they need it then shut it down and go back to using Linux, I'd like to say that that isn't my usage, I use both at the same time, all the time, when my host is running my guest is running, I always have a Linux and Windows desktop both at the same time and wouldn't even think of not using the system that way....think about the possibilities in that type of usage, the connectivity, the productivity. I can do so many things at the same time like playing a game in one monitor on the guest and file maintenance in another on the host system, while keeping up with the forum or surfing the net on another monitor....yes I have 4 monitors.

I think you might see the advantages of such a system.

8

Hm. I have a spare wi-fi card I could use for one of the two. But now I think it might be better to skip getting the 1050ti / passthrough for now, get a second monitor instead, and then re-evaluate the situation down the road. In the mean time I can check how AMD drivers improve on linux with the programs I use. If I can get all my programs working natively under the same driver set there I'll be very tempted to go linux only. Buying a nvidia card down the road in the hope of doing that is another option, and if that fails, hey, now I've got a spare GPU that can be passed through. At least my main hardware will support it and I'll have the ram to do so.

The performance loss by sharing cores/ram isn't a huge deal, obviously it's not as ideal as native, but I think the bigger issues are the screen / ethernet / audio problems. So when I think I can solve all those I'll reconsider.

Going to grab this TN monitor, though I was considering this VA option, as I already have an IPS (older 6 bit iirc) for colours, and want to give freesync a spin.

9

What games and software do you run in your Windows VM? Thoughts on AMD 8 core versus Xeon?

Also, how does it work having host and VM working with passthrough, but only one GPU?

10

I originally built the system so I could run Adobe Photoshop, Illustrator, Corel, and a couple pieces of software for color separation in the printing industry, it worked, in fact worked so well that I have plans to do another computer when it is due for it's upgrade (prolly in 2017).

Gaming-wise, I have played through (start to finish) Fallout 3, Fallout New Vegas, Fallout 4, The entire Half-Life Series, Boarderlands2, Far Cry 3 (playing 4 now) , Witcher 2 (playing now), Metro 2033, there are more but I'm at work and not thinking in that mode...lol

My system is all AMD using a 8370 8 core CPU, 32 gig , and 3 R9 270 GPUs, I think the brand of the CPU means less than having the cores/threads to share and compatible hardware that gives you enough resources so both can be robust...so MB choice is also important.

It doesn't work, you need 2 GPUs one for the host system and one for the guest, you also need two monitors, you need to be able to see both systems at the same time at least in a KVM type of setup, if your talking using something like Virtual box the GPU is passed virtually and you do not have direct access to the hardware from the guest which means most games or programs that want direct access will not work.

In a system like mine KVM/QEMU the hardware that is passed through physically like the GPU isn't ever seen by the host system, it is never available to the host system because it is "blacklisted" at bootup, the host never sees this hardware, doesn't configure it for host use, and that makes it freely available for the guest to use with no conflicts.

Hardware that is virtually passed to the guest ie CPU cores, memory, keyboard (on my system), DVD-ROM, are available to the host when ever the guest isn't running )they are returned to the host system because they were passed virtually not physically), some items like the keyboard and DVD-ROM that I share are available to both systems at the same time meaning both systems will take control and relinquish control of the device so no one really has sole ownership.

It sounds complex and in a way it is, but the reality is that once you have the necessary hardware and jump through a few hoops it works just like two complete systems in the same box that run side by side using and sharing some of the same hardware and resources.

Linux is way more forgiving on hardware than Windows so if your going to run Windows in your guest you have things to consider like I mentioned above in another post, you have to have the resources to give Windows what it needs to run whatever type of software you want and do it stable, the Windows OS itself has pretty big requirements as far as resources to be happy and stable before you even open a program running on top of it.....

You just have to plan it all out, don't scrimp, and don't cut corners, in the end you'll wind up with a kick-ass computer that will do just about anything you ask it to do and do it almost as good as a bare metal install.

Hope this helps.

11

Here's the thing, using the two systems (host & guest) concurrently both needing network access for tasks there will be latency, in file transfers or similar tasks it's not really a big deal, but in playing games like WOW or GW2 that are internet based it will be a killer making the game unplayable, any software that polls the net for whatever reason will cause a problem with a shared NIC and create latency as the control is passed back and forth between the host and guest.

Something you should know is that Nvidia doesn't want you to use their GPUs in a pass through environment, it is very well known that they purposely gimp'd their driver stack to look for virtual environments and error out, there are of course work-arounds that do solve the issue but does come with another performance hit to the guest system.

I'm using the 15.xx drivers and they work fine in the guest, the drivers on the Linux side of things will only get better and better over time.

The performance hit really isn't noticeable to me, Windows is such a quirky OS on the best of hardware and as we all know the performance decreases with every program we install on top of it, the nice thing about running Windows in a KVM is there is really no need for antivirus or firewall software, removing that overhead helps a lot and offsets any performance hit you get from virtualization, you of course can choose to run it but it really isn't necessary because the OS is contained in the KVM container. You also have the ability to take snap-shots of your guest and create new KVMs from that picture in time.

I struggled with the audio latency for several months trying all the fixes that people posted, some helped others did not, in the end a cheap $25 USB sound card solved the problem and since I only need the sound for playing games and wear headphones while playing games it was a perfect fit.

Good luck.

12

You are absolutely right on the need for planning. I did my PCI passthrough sort of on a whim on my existing system, and while it definitely works, and I've been happily using it for everything I've needed from it for about 6 months now, I'm planning on doing a new build soon, and much more planning will be done this time. Like you said, more CPU cores, even more RAM, etc.

Also, +1 for openSUSE as a host. Can't say for any other distro as I haven't tried this on them, but openSUSE has treated me very well as a host.

1 Like
13

I hate to say it but the lack of planning is why so many try the pass through only to fail or have terrible performance, there are enough pitfalls caused by manufactures like Nvidia and Intel along with Asus and other hardware manufactures, and it's hard for people to grasp that Windows isn't a robust environment like Linux is and requires that lots of hardware be available for it to be stable, once you cross that threshold and understand that you are building for two complete systems in one case that shares some resources you get a much better understanding of what is needed.

I use Fedora...it has been very good to me and if I was to build another I'd probably use it again but the flexibility of YaST in OpenSuse makes the pass through much easier to configure and for a first time Linux user would make things easier all the way around because of it's Windows-like operation.

I enjoy my setup so much that I would never consider having Windows on bare metal ever again, there is absolutely nothing software-wise that I have tried that would not run the same as a bare metal install....Windows doesn't know it's in a container at least at this point, MS might change that at some future update but I doubt they would further alienate more users with the number of people that wouldn't even take Win X for free, they are much more likely to embrace virtualization of Windows because it is the future....IMHO

14

Yep, it took me a while to get everything working and all sorted out when I did it, but once everything was working, I never looked back. The performance isn't what it could be, but that's why I'm building a new system in a few months that is tailor-designed for this purpose.

Yeah, it does. Stuff like the boot line can be edited from a GUI for new users.

1 Like
15

I feel the shame when I have to reboot into windows :)

1 Like
16

Have you found anything that would not run? like I said I haven't, I'd agree my performance in some games like Witcher 2 isn't as good as it could be but that is AMD driver based and the fact I'm using older R9 270 GPUs, not the fault of the KVM, I'm really hoping Zen is going to be everything they are saying without the high Intel prices, if it turns out that way I will be building two new machines instead of one....lol

17

That's only because you've learned the virtues of Linux......lol still no shame, we all have Windows programs we are forced to use that we would not if given a choice of comparable software, there just isn't 100% coverage in Linux.

18

Does it only work if the GPUs match? Or could a person put say a R9 390 on passthrough, and something less powerful on the host?

1 Like
19

No it will work with two GPUs that are even of different manufactures, there are people who have used a integrated GPU for the host and a discrete GPU for the guest, like I mentioned above Linux is much more forgiving on the hardware side of things, but it boils down to your intended usage of both the host and guest.

In my case I want to play games on both so need a decent GPU on both sides of the fence, I initially used only two R9 270's in my build but later came up with a 270x card also, I at first use one 270 for the guest and one for the host, when the 270x came along I tried it in the guest side then flipped the cards giving the guest two 270's in crossfire just to see if it was a performance hit or boost.

Today it runs that way the 270x is on the host side and the two 270's are crossfired in the guest, it works well but of course they are outdated cards by today's standards but I don't really care as I'm not a bleeding edge GPU kind of person that needs a 100fps at 4k to be happy. (my biggest gripe is with the AMD driver stack but it is getting better)

There are people who mix AMD in the guest and Nvidia in the host and vise-versa, it's only a issue if you want to use a Nvidia GPU for the guest but it's is doable with a little work, my first attempt at building a KVM was using two old GTX550Ti cards and I got all the way to the point that I needed to load the Nvidia Windows drivers and it error-ed out in Windows and after beating my head against a wall for a few days I found that Nvidia had done that on purpose, it has totally turned me off on their cards.

20

Thanks for all the info! This has clarified a lot of details for me.

My last question on this is about networking.

Is there a solution to this or is multiplayer gaming not an option in the passthrough setup?

1 Like