Welp, lets try that again.
Skylake - Run Windows, and Windows Games, under Linux with UEFI and full video passthrough!
The point of this writeup and these two videos is to show you a Linux box that will literally do everything your windows machine does, but in a nice self-contained way. The reality here is a little less hollywood than waht you see in the video -- Arch Linux is, IMHO, not the best choice for a beginner. The Arch wiki is a stellear set of documentation but it helps if you already know what you are doing. Fedora Rawhide is probably more user friendly as a basis. Still, if you have some Linux experience and have installed Arch successfully, you can probably manage this.
It is worth noting that your goal, if you do this, should be to live on Linux as much as possible. If you want to replicate the Windows (or Mac!) experience on Linux, you're going to be disappointed. Linux works differently, and you should set your expectations as such. You can be perfectly productive on Linux and many games on Steam are already available on Linux natively. You can also run Windows without passing through a graphics card, though you would not be gaming much in that case.
Finally, Wine is another viable alernative that is less "heavy" than what we're doing here. Wine lets you run Windows apps underLinux like native apps. Generally this is a better experience, but games and apps are problematic. The Adobe suite especially problematic, but the Adobe suite will work fine (even with hardware acceleration) under this graphics-card-passthrough methiod.
For this video I used Arch Linux and Kernel 4.1. I would also recommend Fedora Rawhide. It should be possible to do this with Ubuntu or Debian as long as you update to Linux Kernel 4.1 or 4.2. Ubuntu 15.10 is likely to ship with a bleeding edge kernel, and if that happens I will probably update this post. Or perhaps our memebers can :)
Kernel 4.1 improves PCI Passthrough cpabilities and device support. Kernel 4.2/4.3 adds better support for the proprietary/open source driver-in-userspace merger going on right now with the ATI/AMD graphics driver.
This how-to is geared a bit toward Skylake and the Asus Z170 Deluxe. It should work on any board in the channel family (Z170A, etc). It should work on other baords as well, but I haven't yet tested. I plan to, if there is enough interest. It does depend a bit on the motherboard UEFI having proper configuration options for UEFI passthrough, and a functional IOMMU table.
Please note that at this time I am not aware of a working PCI passthrough for the onboard graphics, so this how-to obliviates the possibility of passing through the onboard graphics to a skylake VM.
This how-to is a guide on using a UEFI-based virtual machine (for fast boot times!) and VFIO for PCIe passthrough as opposed to the old ways.
T
he following links were very helpful for me in figuring this out:
http://vfio.blogspot.com/
https://wiki.archlinux.org/index.php/QEMU
https://bbs.archlinux.org/viewtopic.php?id=162768 (reference only, skip to the end, somewhat outdated)
https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMFAt the end of this guide, you should be in a position to boot Windows in a VM in about 3 seconds.
What do you need on the hardware side of things? Ideally:
A UEFI-based graphics card. For our tests we'll be using the Asus Strix 390X. I can't recommend nVidia here because if their graphics driver detects that it is running in a virtual machine, it shuts down. This is old news and has been extensively documented, but it has not been fixed by nVidia. This is bad behavior and you should vote with your wallet.
You do not have to have a UEFI based video card, but in the past one of the reasons that I did not want to do a tutorial on this is that I had a lot of problems with different graphics cards when you startup and shut down the VM. What happens is the Windows drivers leave the card in an undefiend state which can lock the whole machine next time you boot the VM. UEDI is much more well-behaved.
You will want to have a monitor with multiple inputs. You could use a switchbox, but I do not recommend it. The reason is that you will attach both the onboard graphics an the add-in graphics adapter to the same display and toggle between inputs (at least as far as this tutorial goes). You could use two separate displays, if you want, but for this tutorial I am using the A399 4k monitor which is glorious for both productivity and gaming applications.
One use case of your Windows VM also involves steam inhome streaming. In that case, you would not need to attach anything to your onboard graphics card once your virtual machine is setup. You can use remote desktop or VNC to remote into the Windows VM and then Steam on Linux can connect up to Steam on Windows and then game stream from Windows to Linux. This is a bit of a performance hit, obviously, but for many games this is a fine pragmatic solution.
Remote desktop seems to work fine with Adobe OpenCL acceleration which means you can run apps through remote desktop on your virtual machine and this will be a near-native experience again with no physical connection to the video card.
For me, I found it fine to toggle the display input between DisplayPort (Windows) and HDMI (Linux) on the monitor. Do note that I was not able to get 4k/60hz working with Skylake, but it is likely that is resolved in Kernel 4.2.
For control of the virtual machine, for initial setup, I would recommend a second USB keyboard and mouse. You can map these USB peripherals directly through to the virtual machine. Just use lsusb to determine the device and port numbers and substitute those in the quemu command that will appear later. Once you have your VM setup, you can use Synnergy to seamlessly pass mouse and keyboard control between the VMs without having to do any USB mapping.
Synnergy is a program you can install on arch (and download and install on Windows) that uses network sockets and software to "pass" input devices between physical machines. In our case, we'll set it up so that when we go off the left side of the screen in linux, our cursor will appear on the right side of the WIndows display. Scroll Lock will lock the cursor to the display/machine in question. This is also a handy program if you want to, for example, setup your laptop next to your desktop but still only use one set of input devices.
If you have a USB KVM switch, that can also be configured to toggle one keyboard and mouse between different USB ports. Some USB KVM switches are problematic. I would not recommend this IOGEAR KVM because it renumbers the USB devices every time you switch inputs.
Here's our full hardware Rundown:
- Intel i7 6700k Skylake CPU
- Corsair H110i GTX CPU Cooler
- Corsair Vengance LPX DDR4-3200 16gb Memory Kit
- Corsair TX850 PSU
- Intel NVMe 750 SSD (400gb)
- Asus Z170 Deluxe
- Asus STRIX Radeon 390X AMD GOU
- Lian Li PC A51 ATX Case (modded!)--- Hardware Build Vdieo Here:
Getting Started:
The first thing to do is to go into your UEFI and enable VT-d, IOMMU and related options. There should be some coverage of this in the video. As we're really talking about Skylake, the i5 and i7 k parts do have everything you need.
For my setup I gave the evo/lution Arch Linux installer a try, but it didn't know how to deal with my Intel NVMe SSD. So I ended up having to do all that manually to get through the Arch install. Whatever, no big deal.
The next thing to do in UEFI is to make sure that you configure the system to boot from onboard graphics. This means that when you turn the computer on, there should be no output from your add-in graphics card (the Strix 390X in this case) and instead you should see the UEFI and boot screen from attaching to the onboard graphics. I found the Skylake HDMI a bit fiddly with the 4k display, so I used display port for this part. If you aren't running 4k, you aren't likely to encounter an issue with that part. The other issue you may encounter is that Skylake graphics support is not on by default except in kernel 4.3+. You have to enable it on the kernel boot line. Read the rest of this section before proceeding; you may even have to wait before switching the default graphics order to onboard first until after you setup your kernel boot line to support the intel skylake graphics (called i915).
Once you boot up and the boot up screen/UEFI displays from the onboard graphics adapter, the next thing will be to configure Linux not to use the add-in graphics card either. The general way this works is that you blacklist a driver and/or assign an alternative Kernel module (driver) to the device you want to pass through to the VM by PCI ID.
This assumes that you have setup Arch, installed it, downloaded Qemu, libvirt and related virtualization packages. Perhaps some kind soul will supply that list and I'll update this how to with the commands here.
You can use lcpci -ntvv to determine the PCI IDs you want to assign.
If this doesn't work, you will need to blacklist the radeon driver by adding it to the blacklist in /etc -- this sucks though because if you have two Radeon graphics adapters you want to use you really want to specify which card by PCI ID and Bus Location. That's doable, but I'm going to leave that out for now. This can always be updated later with help from the community.
[root@skyfail ~]# lspci -tvnn
-[0000:00]-+-00.0 Intel Corporation Sky Lake Host Bridge/DRAM Registers [8086:191f]
+-01.0-[01]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X] [1002:67b0]
| \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aac8]
+-02.0 Intel Corporation Sky Lake Integrated Graphics [8086:1912]
+-14.0 Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller [8086:a12f]
+-16.0 Intel Corporation Sunrise Point-H CSME HECI #1 [8086:a13a]
+-17.0 Intel Corporation Device [8086:a102]
+-1b.0-[02]----00.0 Intel Corporation PCIe Data Center SSD [8086:0953]
+-1c.0-[03]----00.0 ASMedia Technology Inc. Device [1b21:1242]
+-1c.2-[04-0c]----00.0-[05-0c]--+-01.0-[06]--
| +-02.0-[07]----00.0 Broadcom Corporation BCM4360 802.11ac Wireless Network Adapter [14e4:43a0]
| +-03.0-[08]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612]
| +-04.0-[09]--
| +-05.0-[0a]--
| +-06.0-[0b]----00.0 Intel Corporation I211 Gigabit Network Connection [8086:1539]
| \-07.0-[0c]--
+-1c.4-[0d]----00.0 ASMedia Technology Inc. Device [1b21:1242]
+-1c.6-[0e]----00.0 ASMedia Technology Inc. Device [1b21:1242]
+-1d.0-[0f]--
+-1f.0 Intel Corporation Sunrise Point-H LPC Controller [8086:a145]
+-1f.2 Intel Corporation Sunrise Point-H PMC [8086:a121]
+-1f.3 Intel Corporation Sunrise Point-H HD Audio [8086:a170]
+-1f.4 Intel Corporation Sunrise Point-H SMBus [8086:a123]
\-1f.6 Intel Corporation Ethernet Connection (2) I219-V [8086:15b8]One thing I really like about the Asus Z170 Deluxe is that is has a metric ton of USB3.1 controllers. This is handy because you can pass through an entire PCI USB3.1 controller to your virtual machine and have dedicated USB3.1 ports mapped directly into the virtual machine.
Once you've got your PCI IDs, we need to add them to the boot config and blacklist the radeon driver. I am using systemd to boot my system, but I figure most people are using grub. Fortunately the change is pretty simple -- you just need to add stuff to your kernel line in either case. Before we blacklist the Radeon driver, we need to make sure that the skylake graphics is working fine. That will take some kernel parameters unless you're rocking Kernel 4.3+
You will also need to add some parameters to your kernel for IOMMU and to enable skylake graphics. Weirdly, I had a lot of DRM (render manager, not rights management) errors reported by dmesg until I disabled IOMMU for skylake grapjhics. This should be okay since we don't intend to pass through Skylake Graphics through to the VM but something to be aware of.
Here is the kernel line that worked for me:
options root=PARTUUID=(dont change this part) rw i915.preliminary_hw_support=1 intel_iommu=on intel_iommu=igfx_off pcie_acs_override=downstreamI am not entirely sure I need the PCIe_acs_override=downstream, but I added it anyway.
Once those are set, and you are now booting from onboard graphics from the UEFI to the login prompt, we'll want to disable the Radeon driver:
add
blacklist radeon
to
/modprobe.d/blacklist.conf
echo "blacklist radeon" >> /etc/modprobe.d/blacklist.confFinally, we need to assign those PCI IDs to be usedby the VFIO driver (instead of the now blacklisted Radeon driver). You'll want to pass through tthe PCI IDs of both the Radeon graphics adapter a nd the HDMI audio.
The contents of my /etc/vfio-pci.cfg is
DEVICES="0000:01:00.0 0000:01:00.1"This corresponded to the 390X and its audio component in my system. Yours may vary.
Now, you'll have to reboot once again and you can use lspci to verify that the vfio driver is attached to your add-in graphics card. If that's the case, that's great news.
The next piece of the puzzle is that we want to boot our Virtual Machine in UEFI mode. QEMU/KVM doesn't do that out of the box with Seabios, so we need to get a UEFI bios. Fortunately the Fedora folks have put together an awesome UEFI. You'll want to grab adn install it from here:
There seem to be two UEFI files -- one that is the actual UEFI and one to store UEFI vars. The one to store UEFI vars you'll want to copy to your home folder and use it that way.
Note: You also have to reconfigure some QEMU permissions to access things, or run QEMU as root. This is not ideal to start your QEMU VM as root but is maybe helpful for testing things. I intend to expand this section or perhaps the community can help out here. It is the normal QEMU/KVM stuff for when setting up permissions but it should be documented here. TODO.
Once you've got it installed, you can try to boot up a test virtual machine. I did not use libvirt, an awesome interface for qemu, but you should becuase it simplifies networking. Here is my script:
qemu-system-x86_64 \
-serial none \
-parallel none \
-nodefaults \
-nodefconfig \
-enable-kvm \
-name Windows \
-cpu host,kvm=off,check \
-smp sockets=1,cores=2,threads=1 \
-m 8192 \
-device ich9-usb-uhci3,id=uhci \
-device usb-ehci,id=ehci \
-device nec-usb-xhci,id=xhci \
-rtc base=localtime \
-nographic \
-netdev tap,id=t0,ifname=tap0,script=no,downscript=no -device e1000,netdev=t0,id=nic0 \
-device vfio-pci,host=01:00.0,multifunction=on \
-device vfio-pci,host=01:00.1 \
-vga none \
-device usb-host,bus=ehci.0,hostbus=5,hostport=1 \
-device usb-host,bus=ehci.0,hostbus=5,hostport=1.1 \
-device usb-host,bus=ehci.0,hostbus=5,hostport=1.2 \
-device usb-host,bus=ehci.0,hostbus=5,hostport=1.* \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=./Windows_ovmf_vars_x64.bin \
-hda ./win10.qcow2Note the OVMF code files for the UEFI. More information about that can be found in the reference links but I'd like to hae this how-to be comprehensive. Will update this section later. Or perhaps the community can help out. TODO.
You can simplify the aove qemu commandto just have the device if=pflash commands for UEFI. Witht that, when you start qemu you should see your UEFI graphics card flicker to life and display the UEFI startup screen from the QEMU KVM.
If that worked, from here it is just a matter of putting in an OS Install USB and passing it through to the VM so it can boot off of it. You'll also have to create a permanent storage file for the "hard drive" in the VM or you can pass through a separate physical storage device.
This config passes through 8gb (8192mb) of memory and 2 of the skylake CPU Cores.
Benchmarks of this are coming but so far Gaming speed is 92-101% of native windows speed on the Z170 Deluxe with the 390X.
This is a work in progress. Mods, feel free to fix typos and other corrections. This will probably be turned into a wiki post at some point and I would love to improve this with feedback from the community. This is very much a 0.01 release. I will link the videos here when they are up shortly.