Hey guys, figure I’d do a bit of a write up here since after a few days of on-off work I managed to finally get everything basically working w/ my VFIO setup and I figure I’d just summarize and step through my setup (having lots of experience with Linux, but none with VFIO, KVM, QEMU), as there were a lot of sharp edges I encountered.
My main goal was to be able to use my Valve Index without needing to use a second system or to dual boot (but to potentially have the option of dual booting if I wanted).
There were a lot of twists and turns and research that went nowhere, but here’s a cleaned up version that hits on some of the highlights.
Guide
Hardware
- ASUS ROG Crosshair VIII Hero Wi-Fi X570 (0702 and 0803 BIOS)
- AMD 3700X @ stock (crashes w/ PBO, FCLK 1800; will wait for updates to tweak)
- 4 x 16GB Crucial Ballistix Sport LT 3200@CL16 @ 3200 14-18-18-18-36
- Sabrent 2TB Rocket NVMe 4.0 M.2 (Linux Host)
- Samsung 1TB 960 EVO M.2 (Win10 Guest)
- Sapphire Nitro+ RX 470 4GB (on PCIEX16_1 for Linux Host)
- EVGA GeForce GTX 1080 Ti SC2 (on PCIEX16_2 for Win10 Guest)
- Inateck KT4007 PCI-E USB 3.0 Controller w/ Renasas uPD720201 (on PCIEX16_3 for Win10 Guest)
Software
- Arch Linux (5.2.5-arch1-1-ARCH) Host
- Windows 10 Guest (build 1903)
- qemu-4.0.0-3, libvirt 5.5.0-1, virt-manager 2.2.0-2, edk2-ovmf 20180815-1, ebtables 2.0.10_4-7, dnsmasq 2.80-4, bridge-utils 1.6-3
Motherboard notes
- The ASUS BIOS currently really sucks, it seems to die in lots of ways it shouldn’t, including boot looping when PBO is enabled, and requiring CSM mode to boot with both an RX 470 and RX 570 I tried (I set everything network to Ignore and everything else to UEFI only as bootup is slowed significantly by this)
- I was getting hard shutdowns in mprime - I ended up finding a tip that DRAM Current in the DigiVRM section of the BIOS needs to be set to 130% to prevent this.
- IOMMU groupings is decent (30 groups, every PCIe slot I’ve tried in its own group) however all the USB controllers are “[AMD] Matisse USB 3.0 Host Controller” and in board groups, see USB section for more details
Windows Setup
I set up the Windows drive and GPU first by itself (including AMD Chipset drivers, SteamVR, some initial tuning/testing tools) and then shifted the M.2 over and setup
Dual boot was as simple as copying \EFI\Microsoft\Boot\bootmgfw.efi over to my systemd-boot EFI partition on my new main M.2 once I did the switch.
IOMMU and VFIO
The primary guide I used for setup was Arch Wiki’s PCI passthrough via OVMF
- Enable SVM in AMD CPU Features in the BIOS
- Add kernel parameters
amd_iommu=on iommu=pt - Check
dmesgfor IOMMU groups and use the bash script to check groups (see the above-referenced guide), which worked without issue - Added the GPU and USB card PCI IDs to
/etc/modeprobe.d/vfio.confand the proper modules to/etc/mkinitcpio.confand runmkinitcpio -p linux(orlinux-vfioif you want to manually test first)
Besides discovering the CSM issue w/ my Polaris cards, this part was all pretty straightforward.
OVMF-based Guest VM
This took a bit more fiddling than I would have liked. I had to install a few more packages than was listed:
qemu libvirt ovmf virt-manager ebtables dnsmasq bridge-utils openbsd-netcat
I also ran into issues with the OVMF UEFI loading (you are supposed to be able to specify it in /etc/libvirt/qemu.conf by adding it to the nvram var like:
nvram = [
"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]
But this didn’t work (you should see it loading a Tiano graphic vs SeaBIOS if it is) and I had to learn and fiddle with virsh -c qemu:///system until I could get it right. I ended clearing the nvram setting, using the edk2-ovmf package’s firmware and manually updated my XML (note virsh will autocorrect/parse things on save so if it’s eating settings you need to change things up):
<os>
<type arch='x86_64' machine='pc-q35-4.0'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/guest_VARS.fd</nvram>
<boot dev='hd'/>
</os>
With this I was able to create a q35 VM that uses an existing storage device (pointing to the raw device as SATA - I couldn’t get scsi or virtio, but running CrystalMark gave me 2GB read/writes so I didn’t try too hard to get it booting with the other methods after that). Here’s the XML config, since that part of the setup in the GUI was pretty confusing IMO:
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none' io='native'/>
<source dev='/dev/disk/by-id/nvme-YOUR-DISK-HERE'/>
<target dev='sda' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
I added the GPU/HDMI device, USB board and also the Realtek 2.5Gb (which I didn’t use VFIO for since it doesn’t have drivers by default in the default Arch kernel anyway) as devices to the vm. I’ve actually disabled to the bridged network so that Windows uses the Realtek device as the bridging seems to put a bit of load on my system.
I use the Nvidia workaround so the drivers don’t give guff.
virsh
While setting things up, I found it frequently easier to use virsh. Here’s a little note on accessing system vm’s with virsh. There are probably some important settings I’m forgetting that I changed, although I did try my best to document while I was working on it, so hopefully not too many things…
Windows Activation
Win10 will complain about activation if you set it up for your bare hardware first, but there is a workaround (1, 2) involving using dmidecode to output your information and matching it up. For me, I added something like this, which seemed to work:
<sysinfo type='smbios'>
<bios>
<entry name='vendor'>American Megatrends Inc</entry>
</bios>
<system>
<entry name='manufacturer'>System manufacturer</entry>
<entry name='product'>System Product Name</entry>
<entry name='version'>System Version</entry>
<entry name='uuid'>[YOUR_UUID]</entry>
</system>
<baseBoard>
<entry name='manufacturer'>ASUSTeK COMPUTER INC.</entry>
<entry name='product'>ROG CROSSHAIR VIII HERO (WI-FI)</entry>
<entry name='version'>Rev X.0x</entry>
<entry name='serial'>[YOUR_SERIAL]</entry>
</baseBoard>
</sysinfo>
You should use the dmidecode output to guide you on this.
CPU Topology
I used AMD μProf to help with mapping out my 3700X (and this writeup on CPU-pinning):
./AMDuProfCLI info --cpu-topology
---------------------------------------------
Processor NumaNode Die CCX Core
---------------------------------------------
0 0 0 0 0
0 0 0 0 1
0 0 0 0 2
0 0 0 0 3
0 0 0 0 8
0 0 0 0 9
0 0 0 0 10
0 0 0 0 11
-------------
0 0 0 1 4
0 0 0 1 5
0 0 0 1 6
0 0 0 1 7
0 0 0 1 12
0 0 0 1 13
0 0 0 1 14
0 0 0 1 15
---------------------------------------------
I ended up deciding to use one whole CCX (and 16GB of RAM) for the virtual machine:
<vcpu placement='static'>8</vcpu>
<iothreads>1</iothreads>
<cputune>
<vcpupin vcpu='0' cpuset='4'/>
<vcpupin vcpu='1' cpuset='5'/>
<vcpupin vcpu='2' cpuset='6'/>
<vcpupin vcpu='3' cpuset='7'/>
<vcpupin vcpu='4' cpuset='12'/>
<vcpupin vcpu='5' cpuset='13'/>
<vcpupin vcpu='6' cpuset='14'/>
<vcpupin vcpu='7' cpuset='15'/>
<emulatorpin cpuset='0-1'/>
<iothreadpin iothread='1' cpuset='0-1'/>
</cputune>
GPU and Display
I wasn’t too worried about this initially since I basically am only going to use this for VR, so I just need to be able to launch SteamVR, but early on I bumped up the QXL display adapter’s memory to 32768 in the XML so that I was able to run at a higher resolution.
Because anything that runs on that primary display chugs and it was giving me issues w/ SteamVR Desktop, I ended up disabling the QXL display in Windows entirely and if I have a screen plugged into my 1080Ti it seems to be happier (I plan on a KVM setup; I gave Looking Glass (B1) a try but it was not very stable on my system).
Valve Index Setup
I used a combination of dmesg and lsusb to track down the USB devices that comprise the Valve Index. For those interested:
devices = [
('28de','2613'), # Valve Software Hub
('0424','5744'), # Standard Microsystems Hub
('0424','2744'), # Standard Microsystems Hub
('28de','2102'), # Valve Software VR Radio & HMD Mic
('28de','2300'), # Valve Software Valve Index HMD LHR
('0424','2740'), # Standard Microsystems Hub Controller
('28de','2400'), # Valve Software Etron Technology 3D Camera
]
Before digging out my USB PCI adapters I wrote a script to created custom udev rules to do a virsh attach-device with appropriate hostdev files, but this didn’t work. I didn’t want to go down the ACS path (1, 2), which looked pretty hairy, although I did map out the C8H’s USB controllers (using lsusb -t and plugging things in sequentially, included here for interest:
Rear Front
5 5 3 3 3
5 5 3 3 3
1 1 4
2 1
# Top Left
480M
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
# Back USB-C
10G
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
# Bottom Left
480M
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 004: ID 8087:0029 Intel Corp.
Bus 001 Device 005: ID 0b05:18f3 ASUSTek Computer, Inc.
Bus 001 Device 003: ID 05e3:0610 Genesys Logic, Inc. 4-port hub
# Front Panel USB-C
10G
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
5G
Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub
# Front Panel USB-A, Top Right
480M
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
Bus 003 Device 005: ID 2516:0004 Cooler Master Co., Ltd. Storm QuickFire Rapid Mechanical Keyboard
Bus 003 Device 006: ID 046d:c53f Logitech, Inc.
Instead, I used a separate USB card. I had two, and started with a Fresco Logic card - this actually worked after adding it to VFIO, however it doesn’t reset properly so requires a hard reboot to cycle after a VM start/stop. However, I had a Renasas chipset card as well, which worked fine (here’s a guide, and another discussion) - if you’re looking for a USB card, just do a search for uPD720202 on Amazon or whatever your preferred retailer is (although I am using a 2-port uPD720201 without issues).
Tuning
I was getting some stutters originally (this might have been because I didn’t vfio the USB card originally) but I also went ahead and added <ioapic driver='kvm'/> per this note.
I didn’t do any MSI tuning or switching to invtsc since things seem to be running OK at this point.
TODO/Outstanding:
At this point I’ve played for a few hours in VR, and things seem pretty solid, but there are a few niggles/issues left taht I’ll update as I get through…
libvirtd startup
For whatever reason, on bootup the systemd service has issues so before I run my VM, I need to systemctl restart libvirtd (which runs fine). This could probably be solved by swapping around the service order or something…
KVM Setup
I will attach this to a 4-device KVM - I’ll update with which one I use and if everything works as it should.
UPDATE: I got an AV Access HDMI 2.0 4 port KVM (4K60P YUV444 18Gbps) for $80 which seems to work pretty well. I had some issues with QXL being detected as the primary monitor (maybe due to lag on switching or something) though and in the end, I used virsh to manually remove the QXL adapter entirely from my XML config, which seems to be fine and solves that (a bunch of VR titles get unhappy if they are not connected to the primary display) - note that <graphics> needs to also be removed along with <video> otherwise a video item gets readded (see here).
GPU-Z and Afterburner
Both of these crash my system. I saw some references (1, 2) to adding a kvm.ignore_msrs=1 boot option, but that didn’t fix it for me.
UPDATE: I didn’t knowingly make any changes (updated to kernel 5.2.6 on host?) but this seems to work now. ¯\_(ツ)_/¯
Dual Boot
Somewhere in between where I started and ended, dual booting got borked (Windows says it can’t find the boot media when booting) - I suspect this might have to do with when I installed some virtio drivers trying to get virtio-scsi to work or maybe an UEFI issue. I will need to get motivated enough to poke sometime, as Ryzen Master nor Thaiphoon Burner work in the vm.
That or it’s because I have a Ryzen 2000 still and haven’t updated yet. Shall see.
