Hello World
This is going to be a condensed field manual to guide folks through modern GPU passthrough on arch. Back when I first got into passthrough, it seemed like nobody was doing it and there was no unified guide. But that was back in 2014. Today, there are guides everywhere and video tutorials to accompany the guides. It’s not yet perfect, but we are definitely moving in that direction. That said, lots of guides stop short of all the features I’d like to see in a guide, so I’m going to try to provide a condensed guide that skips the extra discussion and just gets the reader to a complete system. Any questions/suggestions? I’d love to hear em!
Prerequisites
I am going to assume you have a base knowledge of Linux. Arch specifically, but Linux knowledge in general may suffice. You will also need the ability to troubleshoot a broken boot, incorrectly configured kernel command line, and display server errors.
Hardware Recommendations and Requirements
Everyone has their preferred hardware. I’m not going to make any prescriptions. But what I will say is this. Linux support for bleeding edge hardware is a bit less stable than Windows. Passthrough support for bleeding edge hardware is usually unknown until Wendell reports on it. For that reason, I recommend that anyone who is out to build a system specifically for passthrough either clone an existing, known working configuration, or make a build thread to solicit information. If you already have a machine that you want to attempt passthrough on, by all means, give it a try on your hardware, but the fact of the matter is that certain hardware configurations are flat out better than others.
So with that out of the way, what hardware do I recommend? Why, the stack that I’m running, of course. At the moment, there are some nasty bugs with it, but they are easily avoided. Ryzen 5000, Big Navi and B550* have treated me exceptionally well.
General recommendations. Always aim for the highest tier chipset. For AMD, it’s the X series. So X570, X470, etc… Intel, it’s the Z series. For desktop boards anyways. If you’re going HEDT or workstation, that’s only one chipset, it will suffice. I personally prefer AMD at the moment, and find their CPUs have all the features I need.
*AMD B550 seems to work well for me, but if you’re using multiple dGPUs, your mileage may vary.
Software Installation
I’m going to provide a short list of software that you’ll need. Nearly all of it can be found in either the core arch repositories or the AUR. The only one that cannot be reliably found there is OVMF. For some reason, there are a number of OVMF builds that simply do not work for me, so I had to build my own variant. As such, I’ve provided it for download here: ovmf-x64.tar.gz (2.0 MB)
Here’s the list of software you’ll need:
- libvirt
- qemu
- qemu-arch-extra
- virt-manager
- bridge-utils
- dnsmasq
- iptables-nft
System Configuration
Enabling Passthrough | IOMMU
First thing’s first, we need to make sure the IOMMU is enabled. On AMD systems, this is automatic. On intel, you need to pass the kernel flag intel_iommu=on. Make sure IOMMU is enabled in your UEFI. Add this flag to either your /etc/default/grub’s GRUB_CMDLINE_LINUX_DEFAULT entry and regenerate grub’s config or edit the arch.conf entry for systemd-boot. (Highly recommend using systemd-boot)
Once that’s done and you’re rebooted, you need to check that your IOMMU groups will support passthrough. Checking this can easily be done with the iommu bash script.
#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done
we’re looking specifically to see that the GPU and it’s associated components (HDMI audio device, RGB controller, USB controller) are in an IOMMU group by themselves. You’ll be looking for something like this:
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1633]
IOMMU Group 0 01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c0)
IOMMU Group 0 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 0 03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c0)
IOMMU Group 0 03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 0 03:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 0 03:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 1 00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 1 00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge [1022:1634]
IOMMU Group 1 04:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
IOMMU Group 1 04:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
IOMMU Group 1 04:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
IOMMU Group 1 05:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 1 05:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
IOMMU Group 1 06:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 02)
IOMMU Group 1 07:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 1 08:00.0 Non-Volatile memory controller [0108]: ADATA Technology Co., Ltd. XPG SX8200 Pro PCIe Gen3x4 M.2 2280 Solid State Drive [1cc1:8201] (rev 03)
IOMMU Group 2 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge [1022:1632]
IOMMU Group 2 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus [1022:1635]
IOMMU Group 2 09:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [1002:1638] (rev c8)
IOMMU Group 2 09:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1637]
IOMMU Group 2 09:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor [1022:15df]
IOMMU Group 2 09:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU Group 2 09:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Renoir USB 3.1 [1022:1639]
IOMMU Group 2 09:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
IOMMU Group 3 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 51)
IOMMU Group 3 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 4 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166a]
IOMMU Group 4 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166b]
IOMMU Group 4 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166c]
IOMMU Group 4 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166d]
IOMMU Group 4 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166e]
IOMMU Group 4 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:166f]
IOMMU Group 4 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1670]
IOMMU Group 4 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1671]
Note how the 6900xt is in Group 0 by itself. There are other entries, such as Upstream Port and Downstream Port and PCIe bridge, but those devices are just PCIe fabric components, and not actual devices.
WORKAROUND: acs override patch
The ACS override patch can be used to override the IOMMU groups on your motherboard. I will go over the theory in detail at a later time. The ACS patch is unsupported by kernel developers, and can easily cause crashes, because the ACS patch doesn’t actually change IOMMU groups, it just tells the kernel to ignore them. For that reason, I will not be explaining how to use the patch here.
Kernel Driver Overrides | VFIO
Once IOMMU validation is complete, we need to set up kernel driver overrides. Arch, by default, uses mkinitcpio. They are currently in the process of switching to dracut, but I have not yet made that migration. I will publish an addendum to this guide, providing instructions for dracut. Until then, we will use mkinitcpio.
Before you start, you need to determine the vendor and device IDs of the components you are passing through. To find them, you can just look at the output from the iommu command. For me, they are as follows: 1002:73bf,1002:ab28,1002:73a6,1002:73a4. From there, we will insert them into a modprobe configuration file. sudoedit /etc/modprobe.d/vfio.conf
softdep amdgpu pre: vfio vfio_pci vfio-pci
softdep xhci_pci pre: vfio vfio_pci vfio-pci
softdep pcieport pre: vfio vfio_pci vfio-pci
softdep nouveau pre: vfio vfio_pci vfio-pci
softdep nvidia pre: vfio vfio_pci vfio-pci
options vfio_pci ids=
Insert your device ids in a comma separated list on line 6. ex:
options vfio_pci ids=1002:73bf,1002:ab28,1002:73a6,1002:73a4
The first lines are used to tell the kernel that we need to load VFIO drivers prior to any GPU drivers, so we can attach to the GPU we are passing through prior to allowing the real drivers to load.
Once done with that, we need to go into /etc/mkinitcpio.conf and add the modules to the initcpio:
MODULES=(i915 vfio vfio_pci vfio_iommu_type1 vfio_virqfd)
QUIRK:
Whichever GPU drivers you are using on your host should be added before the VFIO drivers in the mkinitcpio list only.
Kernel Driver Configuration | KVM
There are a few flags that need to be set for the kvm kernel modules to provide optimal performance and enable some features, such as sandboxing, to work. To implement these features, create the file /etc/modprobe.d/kvm.conf and insert the following:
options kvm_amd nested=1
options kvm_amd npt=1
options kvm_intel nested=1
options kvm ignore_msrs=1
This enables both nested virtualization and ignoring unknown MRSs. (Model Specific Registers) Nested Virtualization will allow some more performant features of sandboxing to work properly and ignoring MSRs will prevent some hiccups that may occur on specific system calls.
Memory Optimization | HugePages
Linux maps memory in 4kb segments, called pages. Referencing these pages can be slow, so to speed up large memory references, Linux has a feature called HugePages. Hugepages can be either 2mb or 1gb in size, allowing a much smaller page table. This provides a great performance benefit to virtual machines in comparison to using the normal 4kb pages. Normally, I prefer to use 2mb pages because they can be dynamically allocated at runtime. To do this, you use the command sysctl -w vm.nr_hugepages=X In my case, I’m allocating 16GB of RAM to the VM, so I would issue the command: sysctl -w vm.nr_hugepages=8192. Essentially, the number of hugepages should be (memory size in GB)*512.
You can get Linux to boot with this configuration by creating a file in /etc/sysctl.d. I’ve created the following:
/etc/sysctl.d/hugepages.conf
vm.nr_hugepages=8192
Once that’s set, you don’t have to worry about hugepages again.
It is important, however, to note that when memory is allocated as hugepages, almost all normal programs cannot take advantage of these pages. So if you find yourself in need of that memory for a linux task, you can shut down your VM and issue sysctl -w vm.nr_hugepages=0 to re-assign all that memory to normal pages.
Optional: Extra Kernel Performance Optimization | Make Linux Fast Again
The onset of security flaws such as Spectre and Meltdown has required a number of security patches to both the Windows and Linux operating systems. A Windows VM is not aware of the patches having been implemented already at the OS level on the host, so Windows will, by default, implement their own bugfixes to ensure the security of the system. This is good security policy, but some people (like myself) are interested, on some machines, in peak performance, even at the cost of security. Here, we have some options. This forum post provides a list of kernel flags to disable security bugfixes and claw back the performance lost due to speculative execution vulnerabilities on AMD and Intel CPUs. On Windows, it’s a bit more tricky. There isn’t a simple “run fast” kernel flag that you can pass to tell the machine to disable the speculative execution patches. That’s where InSpectre comes in. It digs into the windows registry and makes the changes that you need to disable spectre and meltdown security pathces.
IMPORTANT!
Enabling either of these patches will cause your computer to remain vulnerable to speculative execution. Use this at your own risk.
InSpectre, for the windows VM can be found here: GRC | InSpectre
For the linux host, copy the text at make-linux-fast-again.com into your GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub.
Optional: CPU Affinity for Linux Host | isolcpus
Just like you can set CPU affinity for a process, you can also do something similar for the whole OS. Linux has a kernel flag called isolcpus which prevents the kernel scheduler from assigning tasks to a CPU automatically. This means you can dedicate CPU cores to a specific task. In our case, we are going to use isolcpus to isolate 6 cores (and their associated threads), cores 2 through 6, to the VM.
Adding isolcpus=1,2,3,4,5,6,9,10,11,12,13,14 to your GRUB_CMDLINE_LINUX_DEFAULT variable will isolate CPUs 2 through 7, allowing libvirt exclusive access to them.
Libvirt Configuration | Permissions and Paths
First thing’s first, qemu must run as root. You’ll need to search out the user and group directives, uncomment them and set them to root.
/etc/libvirt/qemu.conf
user = "root"
group = "root"
The OVMF package that I provided can be installed to /usr/local/share/OVMF, to prevent interference with OS packages. This requires some custom configuration to allow libvirt to be able to find the binaries.
To do this, you must edit the nvram list to include our custom OVMF build:
/etc/libvirt/qemu.conf
nvram = [
"/usr/local/share/OVMF/OVMF_CODE.fd:/usr/local/share/OVMF/OVMF_VARS.fd"
]
Once that’s done, we’re ready to define our VM.
VM creation
So, you’ve set all the Host-specific configuration the way you want it, and you’re ready to make the VM? Awesome. Let’s get started.
Libvirt Wizard
In virt-manager, create a new machine. Choose the following options on the wizard:
- Local install media
- Select your win10 iso, tell virt-manager it’s windows 10
- Select your desired CPU threads/memory
- Create a disk image for the virtual machine (40gb, leave default) (this will be changed later, just a placeholder)
- Name it what you want; check “Customize configuration before install”; Choose either Bridge Device or Macvtap device for network
- In overview, select Q35 chipset, UEFI OVMF_CODE firmware.
- In NIC, select device model: virtio
- In SATA disk 1, select disk bus: virtio
- Add hardware; PCI Host Device; add your GPU, HDMI controller, RGB serial controller and USB controller if applicable.
- Begin installation
- Shut down the VM immediately.
Now that’s done, we need to go into the xml file and make some changes. This can be done via either virt-manager or the virsh cli. I don’t care which.
HugePages
Hugepages are a bit of a complex topic that I will go over very briefly. Suffice to say that kernels segregate memory into chunks, or pages. The larger they are, the less overhead of accessing the memory. So, we set up huge pages to improve memory performance. The reason pages aren’t always “huge” is that you can only allocate a whole page, so if your application asks for 4K of memory, but you have your page size set to 4M, the kernel must allocate all 4M to the application. Now, moving on; there are two hugepages. 1GB and 2MB. while 1GB hugepages are best, setup is a bit more complex and frankly, you’ll get perfectly acceptable performance from 2MB hugepages. It’s important to know that once hugepages are allocated, that memory is considered “used”, so you cannot use it for anything else. You can dynamically allocate 2MB hugepages with the command sysctl -w vm.nr_hugepages=$COUNT, but allocating pages after the machine has been running for a while risks fragmentation. It’s best to allow the system to allocate pages early on in the boot process by setting the sysctl variable early. To do this, simply put vm.nr_hugepages=$NUMBER in /etc/sysct.d/hugepages.conf. The value your are setting, $NUMBER, is going to be the amount of memory you gave to the VM, in megabytes, divided by two. So, for my 16GB VM, it would be: 16384/2=8192.
Once this is done, go into your XML with sudo virsh edit $VMNAME and right below the currentMemory entry, add the following:
<memoryBacking>
<hugepages/>
</memoryBacking>
Block Devices
Mounting a block device will vary depending on what type of block device you are using. If you’re using a SATA device, it’s easiest to just use virtio to mount the block device on the VM. In some instances, you can pass the SATA controller through to your VM just like you do the GPU, but that doesn’t always work based on IOMMU groups and other physical limitations of the computer. If you are using NVMe, just pass the PCI device through like you would a GPU.
For SATA, you’ll need to find the disk entry in your xml file and make some edits. So open your vm XML with sudo virsh edit $VMNAME and we’ll get to work.
You’re looking for a block that looks like this:
<disk type='file' device='disk'>
blablabla...
</disk>
We need to make some changes. My configuration looks like this:
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' discard='unmap'/>
<source dev='/dev/sda'/>
<target dev='vda' bus='virtio'/>
</disk>
Only include the discard directive if you are passing through an SSD.
Once this is done, save, exit and launch the vm. libvirt might add an address line in that block, that’s fine and normal.
CPU Pinning
This is another minor optimization that can have major effects on gaming performance. When you set up a VM for the first time, you tell it how many cores to assign it. That just tells qemu to make a pool of X cores to allow the VM to use. This allows the kernel to move tasks to different cores as necessary. Not ideal. Your L1 and L2 cache can disappear at a whim, when the kernel moves the task, and you’ll get hiccups in performance when that happens. So, how do we fix this? CPU Pinning.
There are three main classes of pins that are needed. vcpupin, which is to pin the virtual CPU to a real CPU, emulatorpin, which pins the emulator to a core or set of cores and iothreadpin, which pins the IO threads (used for virtio devices) to a specific core or set of cores.
Jump into your XML with sudo virsh edit $VMNAME and follow along.
This is what my CPU configuration looks like, put it right below your hugepages config:
<vcpu placement='static'>12</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='9'/>
<vcpupin vcpu='2' cpuset='2'/>
<vcpupin vcpu='3' cpuset='10'/>
<vcpupin vcpu='4' cpuset='3'/>
<vcpupin vcpu='5' cpuset='11'/>
<vcpupin vcpu='6' cpuset='4'/>
<vcpupin vcpu='7' cpuset='12'/>
<vcpupin vcpu='8' cpuset='5'/>
<vcpupin vcpu='9' cpuset='13'/>
<vcpupin vcpu='10' cpuset='6'/>
<vcpupin vcpu='11' cpuset='14'/>
<emulatorpin cpuset='7,15'/>
<iothreadpin iothread='1' cpuset='0,8'/>
<iothreadpin iothread='2' cpuset='7,15'/>
</cputune>
Of course, you need to configure your own. You want to make sure you match threads properly to cores.
On Intel CPUs, your threads are paired up even/odd, like so: [0,1] [2,3] [4,5] [6,7]
On AMD CPUs, your threads are paired up with the physical cores first, then all the logical cores. Like so: [0,8] [1,9] [2,10] [3,11] [4,12] [5,13] [6,14] [7,15] for an 8 core CPU.
You can check your CPUs topology with lscpu -e
You’ll also need to configure your CPU emulation settings as well. You need to set the mode to host-passthrough and specify the topology.
It’ll look something like this, the cpu directive should already be there, just search for it and edit:
<cpu mode='host-passthrough' check='partial' migratable='off'>
<topology sockets='1' dies='1' cores='6' threads='2'/>
</cpu>
Optimization Flags
THere are a few hyperv flags that we should really set to optimize the VM. I’m fairly certain there are more, so if you use any that I’ve not included here, just ping me and I’ll be sure to add them.
In features/hyperv, you’ll want the following options:
<relaxed state='on'/>
<vapic state='on'/>
<spinlocks state='on' retries='8191'/>
<vpindex state='on'/>
<synic state='on'/>
<stimer state='on'/>
<reset state='on'/>
<frequencies state='on'/>
Nvidia Error 43
First thing’s first, this is why I do not use nvidia. You should not support a company that intentionally gimps a GPU simply because you’re using an unsupported feature.
Now that’s out of the way, the solutions I used for bypassing E43 on nvidia cards is twofold:
Set the hyperv Vendor ID to something unrecognizable.
<vendor_id state='on' value='C0FEC0FEC0FE'>
set kvm state to hidden.
<kvm>
<hidden state='on'/>
</kvm>
JACK Audio
To get the best software-support for VM Audio, we’re going to use the JACK Audio support that gnif wrote. If you’re building a new system from scratch, it’s best to enable pipewire. If you’re not, we’ll need to switch over. While it’s possible to make this work under JACK directly, you’re then stuck using a much more advanced audio server that is designed for pro audio, rather than consumers. For this reason, I’m using pipewire with the pipewire-jack compatibility layer to support the JACK support here.
Pipewire installation
First thing’s first, we need to get pipewire installed. Arch provides the following packages that we’ll need:
- pipewire
- pipewire-pulse
- pipewire-jack
Additionaly, we need to make sure that pulseaudio and jack are not installed. Remove the following packages if they exist:
- jack
- jack2
- pulseaudio
- pulseaudio-bluetooth
Next, we need to configure pipewire as a drop-in replacement for PulseAudio and JACK:
sudo vim /etc/ld.so.conf.d/pipewire-jack.conf
/usr/lib/pipewire-0.3/jack
Once that’s done, make sure pipewire and it’s compatibility layers are set to launch:
systemctl --user enable --now pipewire
systemctl --user enable --now pipewire-media-session
systemctl --user enable --now pipewire-pulse
If you were running pulse or jack prior to this configuration, it may be ideal to reboot your machine at this point.
VM XML Configuration
For this configuration, I highly recommend using virsh edit to make the changes, because the validator likes to clobber changes if you don’t do it exactly correct.
First change that needs to be done is adding an xmlns flag to the top level domain object. Should look like this when done:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
This will allow us to use some extended features of qemu configuration. In this case, we’re using the qemu:env flag, but we’ll get to that later.
Now let’s configure the audio device. Throw this configuration in your devices xml block:
<sound model='ich9'>
<codec type='micro'/>
<audio id='1'/>
</sound>
<audio id='1' type='jack'>
<input mixingEngine='yes' fixedSettings='yes'>
<settings frequency='48000' channels='2' format='f32'/>
</input>
<output mixingEngine='yes' fixedSettings='yes'>
<settings frequency='48000' channels='2' format='f32'/>
</output>
</audio>
Now that we’ve told QEMU to forward the audio to our jack-pulse configuration, we need to configure a couple environment variables for pipewire to read.
At the very bottom of the xml, right before </domain>, we’ll need to put the following:
<qemu:commandline>
<qemu:env name='PIPEWIRE_RUNTIME_DIR' value='/run/user/1000'/>
<qemu:env name='PIPEWIRE_LATENCY' value='512/48000'/>
</qemu:commandline>
You’ll want to replace the 1000 in /run/user/1000 with your UID. You can get this by running id
Once this is done, save and exit the editor, then open it back up to make sure all the config changes took. I had to fight with the libvirt validator a bit to get these changes to take, but I’ve provided the route that I believe finally got it to take properly.
IVSHMEM (optional)
Now we’re at the fun part. IVSHMEM. Inter-VM Shared Memory. Let’s get started. A bit of pretext. If you are using Looking Glass, you’ll need a way to get audio out of the guest. That’s what the IVSHMEM device is for. It creates a region of memory that is accessable by both the VM and the host. This is how we’ll share the video frames.
Do the math, save the world. Thanks, Mark
Now, let’s set up the IVSHMEM device. Looking glass is going to require some maths. We need to find out exactly how much memory Looking Glass needs, then we must round that number to the nearest power of two.
SHMEM_SIZE = (WIDTH * HEIGHT * 4 * 2) / 1024 / 1024 + 10
So, for example, for a 1080p display, we’ll have the following:
1920 * 1080 * 4 * 2 = 16,588,800
16,588,800 / 1024 / 1024 = 15.82MB + 10 = 25.82MB
So, for 1080p displays, you’ll need to round this up to 32MB.
KVMFR module
To get optimal performance on Looking Glass, it’s recommended to use the KVMFR kernel module. For that, we’ll need to download the latest stable source from Looking Glass - Download Looking Glass, extract it and cd into the resulting directory. From there, we’ll install the module:
cd module
dkms install .
ATTENTION
dkms install .will install the module only for your running kernel. You must install this on all subsequent kernels, as there is no hook for it.
Once the module’s installed, we need to configure it’s permissions with udev:
sudo vim /etc/udev/rules.d/99-kvmfr.rules
SUBSYSTEM=="kvmfr", OWNER="$USER", GROUP="kvm", MODE="0660"
This file will set the proper permissions for both your user and the VM to have access to the kvmfr device node.
Now we need to actually create the kvmfr device.
sudo vim /etc/modules-load.d/kvmfr.conf
kvmfr
Replace the size here with the size that you calculated in the pervious section.
sudo vim /etc/modprobe.d/kvmfr.conf
options kvmfr static_size_mb=32
sudo vim /etc/mkinitcpio.conf
In here, we’ll need to add the kvmfr device to the early load configuration:
MODULES=(kvmfr vfio vfio_pci vfio_iommu_type1 vfio_virqfd)
Once this is done, we’ll need to rebuild our kernel modules:
sudo mkinitcpio -P
VM XML Configuration
Now that we’ve got our kvmfr device set up, we’ll need to teach the VM to see it:
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='ivshmem-plain,id=shmem0,memdev=looking-glass'/>
<qemu:arg value='-object'/>
<qemu:arg value='memory-backend-file,id=looking-glass,mem-path=/dev/kvmfr0,size=32M,share=yes'/>
</qemu:commandline>
You should already have a qemu:commandline section in your XML, just merge the qemu:arg sections into it.
When using Looking Glass, you’ll also need to a spice channel. This will also go into the <devices> section:
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
Now you’re done with VM configuration. Next up is configuring Looking Glass. From here on, we’ll assume you’ve got an OS installed on your VM and you’re ready to go.
Looking Glass
If you want to take advantage of looking glass, there are a number of circumstances that you must take into consideration. The first consideration is the display. Typically, we want to run a Looking Glass VM headless. The problem is that Looking Glass requires a virtual display to work. The recommendation is to either plug a cable into your GPU from a secondary input on your display or buy a DisplayPort Dummy plug or an EDID cloner. Dummy plugs work better for 1080p, however the 4k dummy plugs I’ve found on Amazon and Ebay report very low refresh rates at 1440p (18hz) and 4k (5hz), so I recommend avoiding them for high resolution.
Client
For the Looking Glass client, you must download and build from the source archive on looking-glass.io. Find it here, I recommend the latest stable. Download the source tarball and extract it somewhere in your home directory.
You’ll need the following packages to build the client:
- fontconfig
- libegl
- libgl
- libxi
- libxinerama
- libxss
- nettle
- cmake (make dep)
- fontconfig (make dep)
- spice-protocol (make dep)
- wayland-protocols
If you want the OBS plugin, you’ll need to install obs-studio.
Now let’s build and install the client. If you intend to debug the client, omit -DENABLE_BACKTRACE=no.
mkdir client/build
cd client/build
cmake ../ -DENABLE_BACKTRACE=no -DCMAKE_INSTALL_PREFIX=/usr
make
sudo make install
If you want to build the OBS plugin, do the following from the root of the unzipped tarball:
mkdir obs/build
cd obs/build
cmake ../ -DENABLE_BACKTRACE=no -DCMAKE_INSTALL_PREFIX=/usr
make
sudo install -Dm644 -t /usr/lib/obs-plugins/ ./liblooking-glass-obs.so
Host
Next up is the Looking Glass host. Snag the latest stable version of the host from Looking Glass - Download Looking Glass and follow the installer. Simple as.
Bringing it all together
From here, all you need to do is launch the looking-glass-client. There are some configurations that I recommend, you can put them in an ini file:
vim ~/.looking-glass-client.ini
[app]
shmFile=/dev/kvmfr0
[input]
grabKeyboardOnFocus=no
ignoreWindowsKeys=yes
rawMouse=yes
[spice]
alwaysShowCursor=yes
[egl]
vsync=yes
The only required configuration option for this guide to work is the shmFile flag. Without setting that flag, the Looking Glass client will not know where to find the IVSHMEM device.
Conclusion
This guide took much longer than I planned, due to a number of life situations getting in the way. For those who have been patiently waiting, thank you for your patience.
For those who want to give back, I only ask that you pay it forward. We have a great brain trust here at Level1, but we do not see as much createive or educational activity as we used to. I’d like to get back to that.
Lastly, thanks are in order for the following people:
@Gnif, creator of Looking Glass and the QEMU JACK audio interface, the guy who made passthrough a first-class experience for me, the guy who spent countless hours working on fixing AMD’s broken hardware for them, without so much as a thank you. He deserves a huge round of applause and if you ever have the pleasure of meeting him, buy him a drink.
@Wendell. What can I say? The dude runs multiple businesses and still has time for this community project of Level1. Sometimes I think he doesn’t sleep.
@TheCakeIsNaOH - Creating the initial PR for an NSIS installer for Looking Glass. I know it sounds like a small contribution, but OH BOY, that contribution took the Looking Glass host application from “oh god, I have to go through the setup guide again” to “download, install, done” and small contributions like this should never be forgotten.
@ryedunn - pointing out a few errors an inaccuracies in the guide. Many thanks! 
