Build Advice for GPU Passthru

i have a 770 that seems to work in passthrough, but has micro stutter issues in all ue3 games i played.(fps in the 70's but frametimes would jump all over the place making it feel terrible) to be fair the only ue3 games i tried were smite and bl2.

@Blanger ever experience anything like that on amd cards?..Any amd gpus that work particularly well? happen to know if those 470/480s work better than the fury cards? might have to bookmark this for a future build myself xD

1 Like

I guess to answer your question I should tell you I'm running older R9 270 cards which a lot of people consider to be out dated or the bare minimum in modern performance...lol

Having said that the only real issues I've had micro-stuttering in is Witcher2 which really seems to push my setup, just about everything else I've played has been fine and playable....maybe not on the highest settings but on normal settings graphics-wise I have no issues (I should also mention that in my guest I'm running two 270s in crossfire which has seemed to help)

I do have FPS issues in some games like FarCry3, the FPS do move around a lot between mid 30's to high 60s, I do see some 100fps spikes but not many, I do find it odd that there is a large spread in the FPS but then again I'm not a big time gamer so as long as the game is playable I'm pretty happy.

As to the 470/480s I have no experience so can't advise you, although I do believe they would perform just fine in a pass through situation, no real reason they would not, in my world I'm more concern with providing the KVM with as many resources as I can ie CPU cores, memory, I/O then I am concern with graphic performance, I have not found a game that would not run but like I said I tend to start off on normal settings and tweak the graphic up from there, sometimes I just leave it at normal settings or let the game choose as some games are pretty good at choosing during setup/install.

Honestly I'm as interested as you are about other GPUs and how well they function, once Zen is out next year and if it is everything they say and has a decent price point compared to Intel I may look at building a new box and of course I'd want a newer GPU for that box....

1 Like

thanks mate, i remember linus was having issues with the fury/nano having issues with resets (though the performace seemed good); was just wondering if that was cleared up with the rx stuffs.

Don't mean to hijack the thread or anything, but i do have another question. what settings do you use to give the guest storage? I usually let it write to /dev/sdx directly(this is weird because selinux flips sht and grub adds the vm as a boot option xD), but stuffs like using the host cache and the actual controller it emulates ive never fiddled with. Most of the people that do this dont commit to it; usually just using the image file storage which isnt the best in terms of performance.

1 Like

Actually I use of course a separate drive that has been setup for use by the host system, it will appear in the QEMU/Vert-manager KVM configuration where you choose the space for the KVM to use, in the image below it will appear when you hit the browse button.

e

Then I just allocate the entire drive space to the KVM as in this screen below.

As to the actual path that it is using honestly I've never looked and being at work right now I can't look to see, but the drive does of course show up in the Linux file manager and I can access it from there, but the guest shows it as the full drive that I have allocated with whatever free space the drive has available.

Like you I just accept that QEMU/vert-manager knows what it is doing and roll with it...lol I do not see the KVM as a boot option in grub, I pretty well figured that it was that way because the KVM can't run without the host running under it, but it's interesting that you see it as a boot option.

I did at first use the 40g that it wants to allocate for a VM but once I got a working pass through and a good install of Win 7, I started adding programs and 40g just wasn't even going to come close to what I was going to need, Windows doesn't know it's in a virtual environment and does all the things it does on a bare metal install like a swap file and space allocation, heck 40g with just Windows, a few updates to the OS and a couple games/programs and it will be hitting limits, it's ok to test but not to use as a working OS.

It's why I tell people to look at the specs that the OS and programs need to run on a bare metal install, you need to give that much to the virtual environment for it to have any kind of chance, you would never install Windows on a dual core CPU with 4 gig of ram and a 40g hd and expect it to do very much? and this is what a lot of people try to give the KVM as virtual hardware and come away saying it sucked...of course it did and it would on bare metal also...lol

2 Likes

A follow up.......

I knew there was something else that had to be done to run a Nvidia GPU and I just found it again....


"The GeForce card is nearly as easy, but we first need to work around some of the roadblocks Nvidia has put in place to prevent you from using the hardware you've purchased in the way that you desire (and by my reading conforms to the EULA for their software, but IANAL). For this step we again need to run virsh edit on the VM. Within the section, remove everything between the tags, including the tags themselves. In their place add the following tags:

    <kvm>

      <hidden state='on'/>

    </kvm>

Additionally, within the tag, find the timer named hypervclock, remove the line containing this tag completely. Save and exit the edit session."


I had forgotten about doing this till I just read it again....lol

2 Likes

After a day or 2 of not seeing replies or anything I kinda figured this was a no go thread. Checked my email just a bit ago and seen some messages and came back and YaY! new replies. I've already read them. I've been doing research here and there along with other misc things.

I have read the post over here where you talked and helped a guy over here:


Which while similiar is a bit different.

I have also tried this a while back but found out my motherboard was trash. Space shouldn't be a problem tho since I have an Obsidian 900D. I am pretty much a noob as far as Linux goes. I have done a few things like getting Windows to see files on a Linux PC and playing around with a few programs but I haven't spently nearly the time with Linux as I have with Windows.


Much like you @Blanger I kinda stumbled upon that processor,
http://www.newegg.com/Product/Product.aspx?Item=9SIAAEE3YG0676
which looks like it supports virtualization and I double checked it here:

I eventually stumbled upon this article here:


This kind of led me to check prices on ebay which appears even cheaper. I do know this is a v2 version which seems to be the only one at 2.6GHz. The performance seems to be about the samish as the i7 processor I was looking at earlier.

I would like to know how hot Xeon processors get? I haven't found anything about it..

I am thinking that getting a dual socket motherboard and 2 of these processors might be the way to go tho seem to be having problems with the motherboard side of things.


I thought socket LGA 2011-3 was the same as LGA 2011 only that it allowed higher voltages. It looks like I might be wrong since I have seen some say that a LGA 2011 CPU can't fit into a LGA 2011-3 Socket.

So assuming that LGA 2011 can only work with LGA 2011 socket board I found this board ASRock EP2C602 dual socket LGA 2011 Serverboard:
http://www.newegg.com/Product/Product.aspx?Item=N82E16813157352
I am currently looking at and have been for a few days but I'm still a bit iffy on it. It's the same one in the 16 Core / 32 Thread article. It's so far the cheapest. The Chipset itself is C602 which appears to support Virtualization according to


Altho I'm feeling a bit more comfortable with this board in mind I'm still a bit hesitant because the chipset appears to say it supports PCIe 2.0 while the Newegg page says PCIe 3.0. Also I don't really see where it mentions it supports virtualization. I'm also a bit confused on the RAM too. It talks about DDR3 which I understand. It talks about ECC which I understand... but I see this "R/LR ECC and UDIMM" and I'm a bit lost.

It also doesn't come with audio at all. Which i can kinda understand since it's a server motherboard but then does that mean I'd have to get 2 audio cards? I'm not even sure what to look for in audio cards at all or how I would get 1 speaker on 2 cards.


As for a graphics cards... I'm thinking of picking up an AMD Fury card.. and having that pass thru while my GTX760 runs the Linux environment. A second monitor tho is a no go since I barely have room on my desk for my current one. It looked like in one of the videos a while back that Logan was running a Windows VM in seamless mode. Is that still possible?

Also I read somewhere that the graphics card is suppose to support virtualization too.. which seems confusing. As I understand it AMD supports this in general with the last few generations of their cards but is there a way to be certain about it?


The storage I don't think I'll have much of a problem with. I will pick up an extra SSD for this and might just do a similiar set up and have the Windows VM connect to the Linux guest via share folders / SMB / Samba thingie.

the nvidia workarounds with virsh is if you're using libvirt/virt-manager, and the first ones you listed are more if you're calling qemu via cmd and not using libvirt. Though now that libvirt has support I dunno why anyone wouldn't use virt-manager.

1 Like

Well it's a 150+w CPU so it's going to require a good aftermarket cooling solution, it should run about the same as my 8370 though which is 125w and I'm using a Corsair H100i that works just fine.

While that sounds really cool, you want to make sure the MB has everything you need, most dual socket MBs are "server" class motherboards and some have limited I/O as in USB, SATA, etc, I'd be interested in seeing what you find because that setup would make one killer KVM pass through machine....lol

That MB would be a great choice, Asrock has become one of my favorite manufactures offering lots more bang for the buck and a very good price point.

Yeah.....you could use USB based sound cards or one PCI for the host and a USB for the guest, I use head phones when I play games so the USB sound card was a easy fix for me, they do make switches that will let you input two sources to a set of speakers and switch between the sources....

Nope.....you will need a second monitor, while you can run Windows in a VM using something like VirtualBox and share the same screen you can not with a KVM and pass through....well I say that but sure you could use a KVM (keyboard, video, mouse) switch to switch the inputs between two GPUs to one monitor, it can be done but you really will find that you'll want access to the host screen for one reason or another, a monitor stand, tree, or anything that will increase your area will work, I have 4 monitors on a stand which is 2 27" on the bottom and 2 23-24" on the top if I look at the desk space there isn't enough room for the two 27" monitors to sit side by side but hanging in the air there is room for all 4 monitors, of course I have to tilt my head back to see the upper screens but I don't use them for gaming just viewing stuff.

Nope.....not even an option, anything that will work on bare metal will work in a pass through, it is in theory the same thing since it's physical hardware you are passing through not virtual hardware.

1 Like

the fury/furyx/nano cards have (or used to have not sure if its fixed) a problem where you couldnt reboot the guest without having to reboot the host. Not that big of a problem once you get everything setup, but can be an annoyance.

as far as seemless, not with a gpu passed though.(that i know of) Steam streaming can work, though.

edit: english

1 Like

Don't worry about that, you'll learn quickly with this project. It really gives you a good understanding of how Linux works, at the kernel and driver level.


I almost did the same build, but budget for my upcoming project stopped me. I'd say go for it if you want.

Also, that asrock motherboard looks perfect, if it fits in the 900D. I'm not familiar with the form factor listed, so definitely sanity check that.


Not possible with Passthrough, that's a VirtualBox thing, and it makes use of the virtualbox drivers.

Okay, let's hammer this out once and for all:

Nvidia cards, by their electronic capabilities (the physical properties of the device) support passthrough. Their drivers, within the VM, are intentionally hamstrung to prevent this from working. If Nvidia drivers detect that it's in a VM, they refuse to start, because that's the way Nvidia wants it. This is to try to force you to buy a Quadro. The workaround for this is to disable the KVM and Hyper-V extensions in the VM, which come at a (CPU) performance cost, but the Nvidia drivers will work.

AMD cards, by their electronic capabilities, mostly support passthrough. They have a problem where they don't fully listen to PCIe Bus resets. In laymans terms, this is the same as pressing the reset button on your PC, but only for the GPU. This causes issues with certain cards. (R9 3xx series) Linux developers have claimed to have solved this, but my ASUS R9 380 still encounters this problem, even with Linux 4.8.7. You may or may not have problems with AMD cards. See the only problem here is when you reboot the VM, you have to reboot the host OS as well. So, if you don't reboot the VM, you don't have a problem.


The storage I don't think I'll have much of a problem with. I will pick up an extra SSD for this and might just do a similiar set up and have the Windows VM connect to the Linux guest via share folders / SMB / Samba thingie.

On the topic of Storage, you may want to do something similar to what I've done: I have a ZFS array that hosts my /var, /home, /opt, /srv and has a virtual block device that acts as the C drive for my VM. This allows it to take advantage of the raw storage of my 5x4TB RAID1 array and the speed of my Samsung 850 pro, because that's used as a read/write cache.

It also allows you to take snapshots of the vdev with ZFS as opposed to using QCOW2 snapshots or DD (if you're using direct access to /dev/sdX). On top of that, it's thin-provisioned. (you only use the space that has actual data on it, more or less)

2 Likes

Thank you.....

I wish I had the time to acquire the knowledge to do something like that.....way cool!

This link might shed some light on the topic for you..

1 Like

@blanger That explains a lot of it but didn't see any mention of LR.

@SgtAwesomesauce The board is SSI EEB or something.. anyways it's 12x13 which should fix to my understanding minus a few holes.

1 Like

The LR I believe stands for "registered" memory....nope I was wrong...it stands for "load reducing"

http://www.simmtester.com/page/news/showpubnews.asp?num=167

The terms or nomenclature used in that industry is a moving target....lol

Looks like single 8g sticks of that ram is around $50 per or $100 for 16g 2-stick kits, so memory won't be bad but you have a lot of slots to populate..lol

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Order=BESTMATCH&Description=ddr3++UDIMM&ignorear=0&N=-1&isNodeId=1

It's actually way easier to do than GPU Passthru (the only difficult part is getting ZFS into the kernel, thus the arch package: zfs-dkms). I'm planning on writing a guide for it.

@zyfer Holes can be drilled and tapped. The important thing is that it physically fits without any shorts or bends.

1 Like

Please do.....that is very interesting to me.

Honestly I didn't find the pass through that difficult once I had the right hardware and a basic understanding of what was happening, I'm still a Linux noob but learning more everyday...lol

That's the problem for me, It took me about 6 months to get an understanding of what the hell I'm doing.

I'll get right on it. I'm not great at writing guides (perfectionism gets the best of me), but I'll hopefully have it out within the week.

2 Likes

@Deathright82 Have you seen anything about GTX 1070s with passthru?

@Blanger So with the motherboard I plan on getting, I'm not getting any audio controllers. This means I would need a sound card.

Just recently I found out and tested that I could pipe audio from the graphics card to the monitor and out to my speakers. My current motherboard that I am using a GTX 760 and an ASRock Z75 Pro3 Motherboard. Link to specifics HERE (Hope this HTML stuff works).

Question - does the audio come from from the graphic card itself or is it that the audio controller on the motherboard is just rerouting thru the HDMI out on the graphics card to go thru the monitor thru the speakers?

2 Likes

I tried using the HDMI audio from the passed through card with no success even though as part of passing through the GPU you have to pass the audio portion also, or at least it has to be blacklisted along with the card for it to work, but never got that aspect to work, maybe it would work if the monitor had built-in speakers or a audio out mine did not so yeah I had to use a USB sound card for the guest......if the MB has no audio section you may need two sound cards to have audio from each ( host and guest).

Another option would be if your GPU had a audio out (DIN port) or a video cable that let you "break out" the HDMI audio....I never pursued either.

If I remember correctly HDMI audio from a GPU isn't the greatest quality, not really sure why it's included with computer GPUs except that it is part of the HDMI spec and if you used a TV as a monitor then the audio would work, but a TV has a pre-amp section to boost the audio signal where most computer monitors that have speakers does not so the level and quality is gonna be sub-par compared to a discrete sound card....imho.

1 Like

Afraid I know not of the newer cards from either side.

HDMI audio works to the monitor, sorta. I never could get popping and cracking to stop from it. And had to do http://vfio.blogspot.com/2014/09/vfio-interrupts-and-how-to-coax-windows.html to even get it slightly working as it would occasionally crash the VM.

1 Like