Build Advice for GPU Passthru

Don't worry about that, you'll learn quickly with this project. It really gives you a good understanding of how Linux works, at the kernel and driver level.


I almost did the same build, but budget for my upcoming project stopped me. I'd say go for it if you want.

Also, that asrock motherboard looks perfect, if it fits in the 900D. I'm not familiar with the form factor listed, so definitely sanity check that.


Not possible with Passthrough, that's a VirtualBox thing, and it makes use of the virtualbox drivers.

Okay, let's hammer this out once and for all:

Nvidia cards, by their electronic capabilities (the physical properties of the device) support passthrough. Their drivers, within the VM, are intentionally hamstrung to prevent this from working. If Nvidia drivers detect that it's in a VM, they refuse to start, because that's the way Nvidia wants it. This is to try to force you to buy a Quadro. The workaround for this is to disable the KVM and Hyper-V extensions in the VM, which come at a (CPU) performance cost, but the Nvidia drivers will work.

AMD cards, by their electronic capabilities, mostly support passthrough. They have a problem where they don't fully listen to PCIe Bus resets. In laymans terms, this is the same as pressing the reset button on your PC, but only for the GPU. This causes issues with certain cards. (R9 3xx series) Linux developers have claimed to have solved this, but my ASUS R9 380 still encounters this problem, even with Linux 4.8.7. You may or may not have problems with AMD cards. See the only problem here is when you reboot the VM, you have to reboot the host OS as well. So, if you don't reboot the VM, you don't have a problem.


The storage I don't think I'll have much of a problem with. I will pick up an extra SSD for this and might just do a similiar set up and have the Windows VM connect to the Linux guest via share folders / SMB / Samba thingie.

On the topic of Storage, you may want to do something similar to what I've done: I have a ZFS array that hosts my /var, /home, /opt, /srv and has a virtual block device that acts as the C drive for my VM. This allows it to take advantage of the raw storage of my 5x4TB RAID1 array and the speed of my Samsung 850 pro, because that's used as a read/write cache.

It also allows you to take snapshots of the vdev with ZFS as opposed to using QCOW2 snapshots or DD (if you're using direct access to /dev/sdX). On top of that, it's thin-provisioned. (you only use the space that has actual data on it, more or less)

2 Likes

Thank you.....

I wish I had the time to acquire the knowledge to do something like that.....way cool!

This link might shed some light on the topic for you..

1 Like

@blanger That explains a lot of it but didn't see any mention of LR.

@SgtAwesomesauce The board is SSI EEB or something.. anyways it's 12x13 which should fix to my understanding minus a few holes.

1 Like

The LR I believe stands for "registered" memory....nope I was wrong...it stands for "load reducing"

http://www.simmtester.com/page/news/showpubnews.asp?num=167

The terms or nomenclature used in that industry is a moving target....lol

Looks like single 8g sticks of that ram is around $50 per or $100 for 16g 2-stick kits, so memory won't be bad but you have a lot of slots to populate..lol

http://www.newegg.com/Product/ProductList.aspx?Submit=ENE&DEPA=0&Order=BESTMATCH&Description=ddr3++UDIMM&ignorear=0&N=-1&isNodeId=1

It's actually way easier to do than GPU Passthru (the only difficult part is getting ZFS into the kernel, thus the arch package: zfs-dkms). I'm planning on writing a guide for it.

@zyfer Holes can be drilled and tapped. The important thing is that it physically fits without any shorts or bends.

1 Like

Please do.....that is very interesting to me.

Honestly I didn't find the pass through that difficult once I had the right hardware and a basic understanding of what was happening, I'm still a Linux noob but learning more everyday...lol

That's the problem for me, It took me about 6 months to get an understanding of what the hell I'm doing.

I'll get right on it. I'm not great at writing guides (perfectionism gets the best of me), but I'll hopefully have it out within the week.

2 Likes

@Deathright82 Have you seen anything about GTX 1070s with passthru?

@Blanger So with the motherboard I plan on getting, I'm not getting any audio controllers. This means I would need a sound card.

Just recently I found out and tested that I could pipe audio from the graphics card to the monitor and out to my speakers. My current motherboard that I am using a GTX 760 and an ASRock Z75 Pro3 Motherboard. Link to specifics HERE (Hope this HTML stuff works).

Question - does the audio come from from the graphic card itself or is it that the audio controller on the motherboard is just rerouting thru the HDMI out on the graphics card to go thru the monitor thru the speakers?

2 Likes

I tried using the HDMI audio from the passed through card with no success even though as part of passing through the GPU you have to pass the audio portion also, or at least it has to be blacklisted along with the card for it to work, but never got that aspect to work, maybe it would work if the monitor had built-in speakers or a audio out mine did not so yeah I had to use a USB sound card for the guest......if the MB has no audio section you may need two sound cards to have audio from each ( host and guest).

Another option would be if your GPU had a audio out (DIN port) or a video cable that let you "break out" the HDMI audio....I never pursued either.

If I remember correctly HDMI audio from a GPU isn't the greatest quality, not really sure why it's included with computer GPUs except that it is part of the HDMI spec and if you used a TV as a monitor then the audio would work, but a TV has a pre-amp section to boost the audio signal where most computer monitors that have speakers does not so the level and quality is gonna be sub-par compared to a discrete sound card....imho.

1 Like

Afraid I know not of the newer cards from either side.

HDMI audio works to the monitor, sorta. I never could get popping and cracking to stop from it. And had to do http://vfio.blogspot.com/2014/09/vfio-interrupts-and-how-to-coax-windows.html to even get it slightly working as it would occasionally crash the VM.

1 Like

@Blanger Okie then ty for the info! :D as for sound quality I'm really bad with that in general. Long as it sounds sorta kinda clear and it's loud enough for me to tell what ppl say in a movie that's pretty good for me.

@Deathright82 Mhmm I don't seem to have that problem but will keep the crashing in mind.

1 Like

In my rig, I'm personally using the on-board sound from the motherboard currently, but when I first set the system up, I was using audio from the passed-through GPU. Like @Blanger said, you'll need to blacklist and passthrough both PCI devices more than likely, and I was running HDMI to my monitor which has an audio-out on it, and the sound seemed fine to me...but I'm no audiophile either, so it could have been lousy and I probably wouldn't know the difference...

I am using an R9 390 GPU for what its worth.

1 Like

I should look into that link, I'm not using HDMI audio, my VM is just passing its audio back to the host and I have my speakers plugged into the audio-out on my motherboard, but I have occasionally had my VM crash and it seems related to my audio configuration. I'll have to look into it.

1 Like

The Asrock server board you linked above looks to have ample buss slots to accommodate a couple dual height GPUs (takes up two slots) and have room for a couple PCI sound cards without using a USB card, you have the NICs covered because the MB has two integrated/embedded in the MB.

That will put you pretty far down the road as far as physical hardware that you need to pass through to your guest, most of the rest that you need is optional or your choice in how you want to handle it ie either physically passed or virtually passed to the guest.

I wonder if some of the things I had to turn off to get the Nvidia driver working caused the sound problems? I always figured it was just because of latency with the interrupts, since changing to MSI seemed to improve sound so much. Now I wanna try to fix it. Ever tried to use the virtual audio devices built into qemu? Yeah, the popping was kinda similar to the popping that does.

I also passed through the onboard audio with pretty good results and just used HDMI audio for the host. The onboard audio died shortly after that though. (Probably a coincidence? Had to RMA.)