Best distro for use with VGA passthrough

Hey everyone,

I've been wanting for a while to switch my primary OS to Linux. I made an attempt a few months back after reading a bit about Xen, but I learned that my motherboard does not support AMD IOMMU. I thought I'd try making the switch without Xen-- bad idea; my aging HD4870 had terrible Linux support, resulting in awful performance. I bought a GTX560 to ease my driver problems, but still, being unable to play AAA titles was a deal-breaker. However, it's time I upgrade my motherboard and I've ordered a motherboard that fully supports IOMMU. 

My first question is which distro should I use? I have very limited Linux knowledge; all the distros that I've tried were *buntu based-- that is to say they basically installed themselves. However, I'm sure most suggestions made here will have enough documentation that I could install them without problems. 

As for my second question, I was wondering; can any of you point me in the direction of a guide simple enough that an inexperienced user wouldn't have much trouble following it? It doesn't necessarily have to be Xen, as there could be a better piece of software that I am unaware of. 

I'd be very appreciative of any help you can provide! 

There are no AMD based motherboards from the last 5 years or so that I know off that don't support IOMMU, Asus tries to block it on some boards by either reversing "on" and "off" in BIOS or by only providing the non-blocking BIOS version as "beta" in the hope that people don't dare to use it, but that's pretty well documented on the internet and easy to solve.

AMD HD4870 is supported just fine with the open source R600 driver, in fact, the performance is about on par with the best performance fglrx ever delivered on the HD4/5/6k.

I have AMD RHD7k cards, and I don't use catalyst any more, I use RadeonSI. Users that are on kernel 3.12.5 or newer, should really not use Catalyst any more, there is little advantage in it, unless you need a very specific instruction that is not yet supported for a very specific application. Steam definitely likes the open source linux drivers, with proprietary drivers, whether AMD or nVidia, you have about a 50% chance of having to reinstall your drivers after a Steam update, and those come like every week on linux, whereas the AMD and Intel open source drivers are merged, and Steam has to adapt to them, instead of the other way around, which is a much healthier situation. Unfortunately, nVidia has no functional open source drivers, well, nouveau is functional, but performs very very bad, and nVidia keeps promising, but never keeps it's promises, on the contrary, they keep trying to sabotage open source. I wouldn't buy nVidia if I were you, even on Windows, they have big driver issues, and on linux, the proprietary drivers are virtually unusable in a performance environment, because with every new kernel, the nVidia driver will refuse to compile, and you'll find yourself in Runlevel 3 troubleshooting.

As you only have one graphics card, PCI passthrough is not an option, and frankly, that is the only option I would be comfortable with to write a how-to about (because it's very easy and always works with virt-manager and kvm). VGA passthrough is not easy, it really is something that requires advanced linux knowledge, because you need to script a few things specifically to your system, and you need to know exactly what is on your system, and what the logic behind everything is, and the sequence in which the system deals with the things that are on your system. Sounds vague, but that's just what I can tell you. When you do VGA passthrough, you unbind the graphics card from your linux system, which means you log out of X, but your virtual machine has to keep going, and your host must still allow you to bind the VGA card to the kvm guest. If you use proprietary drivers in your host, unbinding will crash X, and you won't be able to unbind from the guest anyway. And not using proprietary drivers with nVidia cards is masochistic, because the nouveau drivers are so slow.

I've explained most of this before on the forum, why everything is the way it is.

Now as to the question of what linux distro: whatever you want, anything works on anything, virtualization is not a factor in the choice of linux distro, other things may be. Xen, VirtualBox and VMWare, are proprietary commercial solutions, kvm/qemu is not. The commercial solutions offer better paravirtualization, but kvm offers better and faster virtualization, and doesn't break stuff with kernel upgrades.

In other words, use kvm and PCI passthrough. Use your old gpu as a second gpu, dedicate one gpu to linux and one to the windows kvm container.

The open source AMD drivers did work, but I was disappointed with their performance (this very well could have been user error), and the proprietary (legacy) drivers only worked with specific versions of the X server, which, again, I was not happy with. 

I didn't realize that both the host and the VM require a dedicated GPU. I do have two GPUs, one being an HD4870 and the other a GTX560. My PSU is capable of driving both cards... could I use them despite being from different manufacturers?

You should be able to. Since you are effectively using the VGAs in 2 different operating systems. You are going to need to have 2 monitors or a KVM switch to switch between the two VMs

I just switch on the monitor itself, I connect the DVI and the HDMI cables, one goes to one card, the other to the other. I'm not even switching, I've scripted xrandr to switch off the output of the host card when I move to the virtual desktop that runs the VM in full-screen, and have made that priority on the monitor, so it return to that input when it comes back on. I often also use multiple monitors, one for each desktop. There are plenty of solutions without the hassle of a video input switcher, but some people like a video switcher, my wife uses one, she prefers that solution over others, but then she uses her Windows guest more than I do, I haven't used Windows in over 6 months to be honest.

Since you cards are running in different virtual systems and your main Linux install doesn't have to deal with one of them, there shouldn't be a problem.

Here's Zoltan's how-to on the subject: What if I want Everything?


As for the OSS Radeon drivers: never got those to work, either. They just don't work for pretty much anything I have besides providing basic functionality, and I've come to accept that.

There are two AMD OSS drivers: the R600 and the RadeonSi. The R600 delivers more than 80% of the performance of Catalyst, and are the best choice for cards up to RHD6k series. The RadeonSi is still in development, but is getting better literally every week, and only works well with kernels that have merged the functionality of the RHD7k and R series. If you use an old distro, that is not up-to-snuff with kernel development, you will never get good performance out of the RadeonSi, until that distro catches up, which can take a very long time. If you insist on using an older distro, like Ubuntu 13.10 for instance, you can still compile a newer kernel yourself from git, but expect extreme breakage with Unity and XMir. This problem does not exist with community Ubuntu distros, i.e. non-Unity Ubuntus.

There is however a solution for Unity Ubuntu users, there is a community repo somewhere that provides packages that allow for installation of the kernel 3.12.5 first build, which is not great but acceptable, and that prevents breakage. I don't remember the link, but I think it was an Australian project. I tried it when I tested Ubuntu 13.10, and it worked.

There will always be one point of low performance with the OSS drivers, and that's Adobe flash. This is not a driver problem, but a deliberate blockage by Adobe. The way Adobe does it, is quite crazy: when you're running proprietary AMD or nVidia drivers in linux, Adobe doesn't let flash use the GPU, so there is a lot of load on the CPU. This is known issue, and Adobe has made clear that they want it that way. On the other hand, when you use an OSS driver, Adobe prevents flash from even using the GPU, AND limits the CPU activity, so that you get the same flash performance out of an Atom N450 and an i7-4770. There is a fix for it, but it involves a little hack, and most users that use OSS drivers, will not use flash anyway, but HTML5 instead, and that works just the same with OSS drivers and proprietary drivers. So it's a non-issue really.

In games, the AMD OSS drivers perform quite well. There is still some work to be done on the RadeonSi driver, but most of the lack of performance has been solved with the linux kernel 3.12.5-6-7, and almost all of it is solved by kernel 3.13. I use kernel 3.13 on a fedora install for gaming with RadeonSi, and the performance of the GPU in comparison to kernel 3.12 first build has all but doubled.

It's one of the reasons I recommend bleeding edge distros, this year, HSA comes out in linux, and the full HSA implementation in fedora release will be for August/September, depending on how fast Intel can sort its problems (and they still have huge problems), because fedora will wait for Intel to at least have a functional version of Beignet, even if it's limited, and for Intel to at least patch the huge graphics performance regression on modern kernels, but chances are, that it's a design flaw in Haswell (as I posted before Haswell was released, I had a netburst-feeling for Haswell, I think that feeling was right), and that Intel will release Broadwell in summer to offer a solution that is a bit of a closer match for AMD's amazing APU line-up, even though I believe that Intel will need about two years and another CPU-generation to catch up to Kaveri and Kabini, because even if Broadwell matches Kaveri, which is not very likely because Intel lacks the graphics experience, AMD can downstream Kabini units to Kaveri, and offer hexa- or octocore APU's, because Kabini proves that they have them ready, and they're just moving forward with the parking brake on for the moment (AMD also sells Intel CPU's, they have no interest in deteriorating the relationship with Intel, which has been quite good since 2009). People that use bleeding edge distros, will gain at least 6 to 12 months of early adopter benefit, as bleeding edge distros have everything laid out, tested and stable by summer, whereas non bleeding edge distros will only be able to start testing and integrating HSA for the October to December 2014 development releases.  Ubuntu 14.04 is an LTS release, but will have to pass on HSA completely, and that will knock Ubuntu back considerably. 14.10 is just a development release, which have been getting hugely unstable recently, to the point of being very hard to use, and even if it's stable, Canonical will not be able to provide HSA integration by October, so Ubuntu users will have to wait at least for the 15.04 development release to even start trying HSA, and it will be hugely unstable, so by the time Canonical gets some stability in the implementation of HSA, Fedora, Gentoo and Arch users, will be using HSA in a stable manner for a year already. That's 25-33% of the life cycle of the hardware that it's running on, that's a lot of missed opportunity and investment write-off.

Is the possibility for VGA passthrough with just Intel graphics real? I would like to do a video passthrough on my ThinkPad X200s.

You say that one VGA card should be left for the host; is that really so? Servers without a VGA card are very real after all.

Yes it can be done with just one card, as I've explained before, through either one of two solutions:

1. VGA passthrough instead of PCI passthrough;

2. Checking the box in virt-manager that the guest should start automatically when the host boots up (requires autologin in the host though), if you're only going to use the guest OS.

"Headless" servers are indeed very real, but they don't run games. I don't think you'd like the experience of gaming in an ssh session lolz

Zoltan is it possible to install Linux with out a graphics card plugged in and using the on board graphics on your CPU, then plug in the graphics card in (with monitor cable still connected to your motherboard) and add it as a PCI device in virt-manager? Would it act the right way?

Yup, just use a distro with an automated hardware configuration tool. (Have automated config tools out of the box: Fedora, OpenSuSE, Mageia, Rosa, Manjaro, Sabayon. Manjaro, Mageia and OpenSuSE are arguably the best for doing this with limited knowledge of the linux base system, especially Manjaro and Mageia as they have very easy to use CLI tools (mhwd and drakxconf respectively) that allow you to check and configure everything to your liking even if your X server has crashed because of the graphics card shenanigans (that's about the worst that could happen, and it's not that probable even, but you might want to check if your system is going to survive the abuse of hotplugging the single most energy-hungry device in your case). Ubuntu and derivatives do not have automated configuration tools, but they have semi-automated tools, they won't configure automatically but will still be pretty easy to configure, as long as X is running. Pretty much everything else will have to be configured with the traditional tools in CLI (not that hard, basically editing two or three text files and running a couple of commands).

I would not hotplug the graphics adapter if I were you, just out of respect for the hardware. What you can do is disable the graphics adapter in either BIOS or linux itself, configure the iGPU as main adapter, and then reenable the other adapter.

But basically I don't really see your problem though to be honest. The right thing to do is to set the iGPU as the primary adapter in BIOS, then unbind the discrete GPU and bind it to the guest. You do have to unbind the PCI slot from the host before being able to bind it to the guest.

Right I think that's what I want to do but I'm how sure how to do it.