Zen2 (Ryzen 3700X) and X570 (ASUS C8H) Win 10 VR on Arch Linux

Hey guys, figure I’d do a bit of a write up here since after a few days of on-off work I managed to finally get everything basically working w/ my VFIO setup and I figure I’d just summarize and step through my setup (having lots of experience with Linux, but none with VFIO, KVM, QEMU), as there were a lot of sharp edges I encountered.

My main goal was to be able to use my Valve Index without needing to use a second system or to dual boot (but to potentially have the option of dual booting if I wanted).

There were a lot of twists and turns and research that went nowhere, but here’s a cleaned up version that hits on some of the highlights.

Guide

Hardware

  • ASUS ROG Crosshair VIII Hero Wi-Fi X570 (0702 and 0803 BIOS)
  • AMD 3700X @ stock (crashes w/ PBO, FCLK 1800; will wait for updates to tweak)
  • 4 x 16GB Crucial Ballistix Sport LT [email protected] @ 3200 14-18-18-18-36
  • Sabrent 2TB Rocket NVMe 4.0 M.2 (Linux Host)
  • Samsung 1TB 960 EVO M.2 (Win10 Guest)
  • Sapphire Nitro+ RX 470 4GB (on PCIEX16_1 for Linux Host)
  • EVGA GeForce GTX 1080 Ti SC2 (on PCIEX16_2 for Win10 Guest)
  • Inateck KT4007 PCI-E USB 3.0 Controller w/ Renasas uPD720201 (on PCIEX16_3 for Win10 Guest)

Software

  • Arch Linux (5.2.5-arch1-1-ARCH) Host
  • Windows 10 Guest (build 1903)
  • qemu-4.0.0-3, libvirt 5.5.0-1, virt-manager 2.2.0-2, edk2-ovmf 20180815-1, ebtables 2.0.10_4-7, dnsmasq 2.80-4, bridge-utils 1.6-3

Motherboard notes

  • The ASUS BIOS currently really sucks, it seems to die in lots of ways it shouldn’t, including boot looping when PBO is enabled, and requiring CSM mode to boot with both an RX 470 and RX 570 I tried (I set everything network to Ignore and everything else to UEFI only as bootup is slowed significantly by this)
  • I was getting hard shutdowns in mprime - I ended up finding a tip that DRAM Current in the DigiVRM section of the BIOS needs to be set to 130% to prevent this.
  • IOMMU groupings is decent (30 groups, every PCIe slot I’ve tried in its own group) however all the USB controllers are “[AMD] Matisse USB 3.0 Host Controller” and in board groups, see USB section for more details

Windows Setup

I set up the Windows drive and GPU first by itself (including AMD Chipset drivers, SteamVR, some initial tuning/testing tools) and then shifted the M.2 over and setup

Dual boot was as simple as copying \EFI\Microsoft\Boot\bootmgfw.efi over to my systemd-boot EFI partition on my new main M.2 once I did the switch.

IOMMU and VFIO

The primary guide I used for setup was Arch Wiki’s PCI passthrough via OVMF

  • Enable SVM in AMD CPU Features in the BIOS
  • Add kernel parameters amd_iommu=on iommu=pt
  • Check dmesg for IOMMU groups and use the bash script to check groups (see the above-referenced guide), which worked without issue
  • Added the GPU and USB card PCI IDs to /etc/modeprobe.d/vfio.conf and the proper modules to /etc/mkinitcpio.conf and run mkinitcpio -p linux (or linux-vfio if you want to manually test first)

Besides discovering the CSM issue w/ my Polaris cards, this part was all pretty straightforward.

OVMF-based Guest VM

This took a bit more fiddling than I would have liked. I had to install a few more packages than was listed:

qemu libvirt ovmf virt-manager ebtables dnsmasq bridge-utils openbsd-netcat

I also ran into issues with the OVMF UEFI loading (you are supposed to be able to specify it in /etc/libvirt/qemu.conf by adding it to the nvram var like:

nvram = [
	"/usr/share/ovmf/x64/OVMF_CODE.fd:/usr/share/ovmf/x64/OVMF_VARS.fd"
]

But this didn’t work (you should see it loading a Tiano graphic vs SeaBIOS if it is) and I had to learn and fiddle with virsh -c qemu:///system until I could get it right. I ended clearing the nvram setting, using the edk2-ovmf package’s firmware and manually updated my XML (note virsh will autocorrect/parse things on save so if it’s eating settings you need to change things up):

 <os>
    <type arch='x86_64' machine='pc-q35-4.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2/ovmf/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/guest_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>

With this I was able to create a q35 VM that uses an existing storage device (pointing to the raw device as SATA - I couldn’t get scsi or virtio, but running CrystalMark gave me 2GB read/writes so I didn’t try too hard to get it booting with the other methods after that). Here’s the XML config, since that part of the setup in the GUI was pretty confusing IMO:

<disk type='block' device='disk'>
    <driver name='qemu' type='raw' cache='none' io='native'/>
    <source dev='/dev/disk/by-id/nvme-YOUR-DISK-HERE'/>
    <target dev='sda' bus='sata'/>
    <address type='drive' controller='0' bus='0' target='0' unit='0'/>
 </disk>

I added the GPU/HDMI device, USB board and also the Realtek 2.5Gb (which I didn’t use VFIO for since it doesn’t have drivers by default in the default Arch kernel anyway) as devices to the vm. I’ve actually disabled to the bridged network so that Windows uses the Realtek device as the bridging seems to put a bit of load on my system.

I use the Nvidia workaround so the drivers don’t give guff.

virsh

While setting things up, I found it frequently easier to use virsh. Here’s a little note on accessing system vm’s with virsh. There are probably some important settings I’m forgetting that I changed, although I did try my best to document while I was working on it, so hopefully not too many things…

Windows Activation

Win10 will complain about activation if you set it up for your bare hardware first, but there is a workaround (1, 2) involving using dmidecode to output your information and matching it up. For me, I added something like this, which seemed to work:

<sysinfo type='smbios'>
    <bios>
      <entry name='vendor'>American Megatrends Inc</entry>
    </bios>
    <system>
      <entry name='manufacturer'>System manufacturer</entry>
      <entry name='product'>System Product Name</entry>
      <entry name='version'>System Version</entry>
      <entry name='uuid'>[YOUR_UUID]</entry>
    </system>
    <baseBoard>
      <entry name='manufacturer'>ASUSTeK COMPUTER INC.</entry>
      <entry name='product'>ROG CROSSHAIR VIII HERO (WI-FI)</entry>
      <entry name='version'>Rev X.0x</entry>
      <entry name='serial'>[YOUR_SERIAL]</entry>
    </baseBoard>
  </sysinfo>

You should use the dmidecode output to guide you on this.

CPU Topology

I used AMD μProf to help with mapping out my 3700X (and this writeup on CPU-pinning):

./AMDuProfCLI info --cpu-topology
---------------------------------------------
 Processor  NumaNode     Die    CCX      Core
---------------------------------------------
   0         0           0      0        0   
   0         0           0      0        1   
   0         0           0      0        2   
   0         0           0      0        3   
   0         0           0      0        8   
   0         0           0      0        9   
   0         0           0      0        10  
   0         0           0      0        11  
                                -------------
   0         0           0      1        4   
   0         0           0      1        5   
   0         0           0      1        6   
   0         0           0      1        7   
   0         0           0      1        12  
   0         0           0      1        13  
   0         0           0      1        14  
   0         0           0      1        15  
---------------------------------------------

I ended up deciding to use one whole CCX (and 16GB of RAM) for the virtual machine:

<vcpu placement='static'>8</vcpu>
<iothreads>1</iothreads>   
<cputune>     
   <vcpupin vcpu='0' cpuset='4'/>     
   <vcpupin vcpu='1' cpuset='5'/>     
   <vcpupin vcpu='2' cpuset='6'/>     
   <vcpupin vcpu='3' cpuset='7'/>     
   <vcpupin vcpu='4' cpuset='12'/>     
   <vcpupin vcpu='5' cpuset='13'/>     
   <vcpupin vcpu='6' cpuset='14'/>     
   <vcpupin vcpu='7' cpuset='15'/>     
   <emulatorpin cpuset='0-1'/>     
   <iothreadpin iothread='1' cpuset='0-1'/>   
</cputune>

GPU and Display

I wasn’t too worried about this initially since I basically am only going to use this for VR, so I just need to be able to launch SteamVR, but early on I bumped up the QXL display adapter’s memory to 32768 in the XML so that I was able to run at a higher resolution.

Because anything that runs on that primary display chugs and it was giving me issues w/ SteamVR Desktop, I ended up disabling the QXL display in Windows entirely and if I have a screen plugged into my 1080Ti it seems to be happier (I plan on a KVM setup; I gave Looking Glass (B1) a try but it was not very stable on my system).

Valve Index Setup

I used a combination of dmesg and lsusb to track down the USB devices that comprise the Valve Index. For those interested:

devices = [
  ('28de','2613'), # Valve Software Hub
  ('0424','5744'), # Standard Microsystems Hub
  ('0424','2744'), # Standard Microsystems Hub
  ('28de','2102'), # Valve Software VR Radio & HMD Mic
  ('28de','2300'), # Valve Software Valve Index HMD LHR
  ('0424','2740'), # Standard Microsystems Hub Controller
  ('28de','2400'), # Valve Software Etron Technology 3D Camera
]

Before digging out my USB PCI adapters I wrote a script to created custom udev rules to do a virsh attach-device with appropriate hostdev files, but this didn’t work. I didn’t want to go down the ACS path (1, 2), which looked pretty hairy, although I did map out the C8H’s USB controllers (using lsusb -t and plugging things in sequentially, included here for interest:

Rear     Front
5 5 3 3  3
5 5 3 3  3
1 1      4
2 1

# Top Left
480M
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

# Back USB-C
10G
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub

# Bottom Left
480M
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  Bus 001 Device 004: ID 8087:0029 Intel Corp. 
  Bus 001 Device 005: ID 0b05:18f3 ASUSTek Computer, Inc. 
  Bus 001 Device 003: ID 05e3:0610 Genesys Logic, Inc. 4-port hub

# Front Panel USB-C
10G
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  5G
  Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. ASM1074 SuperSpeed hub

# Front Panel USB-A, Top Right
480M
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  Bus 003 Device 002: ID 174c:2074 ASMedia Technology Inc. ASM1074 High-Speed hub
  Bus 003 Device 005: ID 2516:0004 Cooler Master Co., Ltd. Storm QuickFire Rapid Mechanical Keyboard
  Bus 003 Device 006: ID 046d:c53f Logitech, Inc. 

Instead, I used a separate USB card. I had two, and started with a Fresco Logic card - this actually worked after adding it to VFIO, however it doesn’t reset properly so requires a hard reboot to cycle after a VM start/stop. However, I had a Renasas chipset card as well, which worked fine (here’s a guide, and another discussion) - if you’re looking for a USB card, just do a search for uPD720202 on Amazon or whatever your preferred retailer is (although I am using a 2-port uPD720201 without issues).

Tuning

I was getting some stutters originally (this might have been because I didn’t vfio the USB card originally) but I also went ahead and added <ioapic driver='kvm'/> per this note.
I didn’t do any MSI tuning or switching to invtsc since things seem to be running OK at this point.

TODO/Outstanding:

At this point I’ve played for a few hours in VR, and things seem pretty solid, but there are a few niggles/issues left taht I’ll update as I get through…

libvirtd startup

For whatever reason, on bootup the systemd service has issues so before I run my VM, I need to systemctl restart libvirtd (which runs fine). This could probably be solved by swapping around the service order or something…

KVM Setup

I will attach this to a 4-device KVM - I’ll update with which one I use and if everything works as it should.

UPDATE: I got an AV Access HDMI 2.0 4 port KVM (4K60P YUV444 18Gbps) for $80 which seems to work pretty well. I had some issues with QXL being detected as the primary monitor (maybe due to lag on switching or something) though and in the end, I used virsh to manually remove the QXL adapter entirely from my XML config, which seems to be fine and solves that (a bunch of VR titles get unhappy if they are not connected to the primary display) - note that <graphics> needs to also be removed along with <video> otherwise a video item gets readded (see here).

GPU-Z and Afterburner

Both of these crash my system. I saw some references (1, 2) to adding a kvm.ignore_msrs=1 boot option, but that didn’t fix it for me.

UPDATE: I didn’t knowingly make any changes (updated to kernel 5.2.6 on host?) but this seems to work now. ¯\_(ツ)_/¯

Dual Boot

Somewhere in between where I started and ended, dual booting got borked (Windows says it can’t find the boot media when booting) - I suspect this might have to do with when I installed some virtio drivers trying to get virtio-scsi to work or maybe an UEFI issue. I will need to get motivated enough to poke sometime, as Ryzen Master nor Thaiphoon Burner work in the vm.

4 Likes

Well the fact that you passed through devices on a Bios with the ‘abb’ version of Agesa brings me hope that I’ll be able to as well… just as soon as my board gets that update. But I don’t even have ‘ab’ yet. I really hope I don’t wait a week or more and get ‘ab’ instead of ‘abb’…

My experience with X570 was the opposite of yours. I had to turn CSM off to even get video with both GPUs installed. But maybe I can look forward to contrary behavior with the new bios as well. :roll_eyes: That or it’s because I have a Ryzen 2000 still and haven’t updated yet. Shall see.

I have a Renasas usb card, not sure if it’s the uPD720202, but I couldn’t use it with my horrible IOMMU groups on the X470 board I replaced. Once I get my GPU passing again I’ll have to see if I get lucky there.

I believe you have to be using a virtio-scsi controller with your drives to take advantage of iothreads as you have defined in your cpu pinning, but that is just what I read in a redhat page I believe, I hadn’t confirmed it before I ran into a roadblock on my previous board that resulted in me moving to X570. That and the disk needs to be set as

<driver  name='' type='' cache='' io='threads'/>

Well not literally. You get the idea.

Err… now that I look at my xml config I remember I abandoned trying to setup a virtio-scsi device to connect my drives to because I realized that you could just set qcow2 files to use iothreads but it refuses to run if I set cache to ‘none’. Isn’t all this nice and straight forward?

Might have to apply that bios fakery to my VM when I start back at. I didn’t have any activation issues, but it might have been because I based my config off one created from the Virt-manager wizard? Still it might help prevent future trouble as I modify my hardware, both real and virtualized, in the VM.

1 Like

This is an awesome write up, thank you for posting it. It’s nice to see the people who made passthrough work rather then just the people who are stuck.

1 Like

I havent had this problem but I have had issues getting it to boost correctly. A lot of my stability issues on this board are actually ram related.

What voltage are you running ram at?

My apologies for being stuck then :stuck_out_tongue:

Fwiw I had it working before I was forced to change out my MB.

I will say that while this setup took a fair bit of linux/google-fu as there are plenty of sharp edges and not much consolidated information (also, IMO the Virtual Machine Manager documentation is really bad for someone who’s never touched it before and the storage pool stuff is very unclear), a lot of it also comes down to being able to use the right hardware.

I could see this being hard to impossible trying to pass-through an AMD card, relying on ACS for a board with poor IOMMU groupings, or having a USB PCI chipset that doesn’t support proper resetting, etc. There also seems to be a very long tail of instability and problems, and also a lot of out of date forum threads.

But I am pretty happy that I managed to get it all (mostly) working and hope my streamlined writeup (removed lots of blind alleys from my notes) helps someone with a similar setup.

I am running my RAM at 1.40V atm (seemed ok at 1.38V and at 1.45V as well), but haven’t spent a lot of time tweaking. Even with very loose timings, setting IF to 1800 seems to fail to POST. I actually bought a set of HyperX Predator 4x16GB 3200CL16s (HX432C16PB3AK4/64) to try out as they were on ASUS’s 4 DIMM QVL, but they actually didn’t clock any better than my Ballistix kit (once I figured out that increasing DRAM Current was necessary for Ballistix stability, although the HyperX did run mprime without requiring the current bump).

Due to not having Thaiphoon Burner, Ryzen DRAM Calculator, or Ryzen Master on Linux, and having slow boot-up times due to CSM I’m leaving well enough alone for now.

While I do think that the C8H has some issues w/ 4-sticks of RAM, I suspect its CPU boost/voltage wonkiness is independent (eg from Stilt’s reporting on actual vs reported voltage). I’ve yet to have the appetite to go poking around in the options at least until they’ve ironed out the overvoltage issues. (I was able to get the nct6798 reporting somewhat sensibly in Linux,but honestly, I’d like to have the k10temp support at least (5.4 kernel?) before tweaking.)

I suspect that is the case as well.

Im running at 18-18-18-38 3600 on b-die, at 1800 FCLK on 1.4v though I suspect it would work fine at 1.35. I’m going to give the dram calculator a go and see if I can get down to CL14 3200 since thats theoretically faster.

Sorry to hijack your thread. The experience on this board seems to be hit and miss.

I’m in the same boat as you; glad I’m not the only one having trouble with X570 (I’m also on using Ryzen 2000).

I had to disable CSM to get both cards recognized at all, but right now my VM fails to start with “stuck in D3” message.

What error(s) are you stuck on?
Oh, I see you’re the same person from this other post, and you have the same “stuck in D3” problem that I do. forum. level1techs. com/t/gpu-passthrough-advice-on-my-boot-failure-issue-mobo-gpu-combo/145861/5
Let me know if you make any progress (and I’ll do likewise : )

Aside: I put details of my (non-) progress here, in case you’re interested: reddit: /r/VFIO/comments/csr7e5/x570_futile_for_now/

Also got my VM to work a few days ago. Same processor, with a passed-through 5700 XT. Can’t restart the VM because of the reset bug though (“Unknown PCI header type 127”), but hopes are high to have gnif figuring out a solution.

Had the same problem. For me it kept complaining about a missing /etc/kvm folder (systemctl status libvirtd.service). Ended up just creating a timer as workaround.
I Added this file:

# /etc/systemd/system/libvirtd.timer
[Unit]
Description=Stops complains about missing /etc/kvm

[Timer]
OnBootSec=6

[Install]
WantedBy=basic.target

Then disabled libvirtd.service and enabled libvirtd.timer via systemctl. Works for me, I’d really have to rush to notice those 6 extra seconds.

May look into your CPU solution at some time. I was quite lazy and just straight-forward passed through 7 whole cores.

  <cpu mode="host-passthrough" check="partial">
    <topology sockets="1" cores="7" threads="2"/>
  </cpu>

Got great performance with this, though yours may be better. It’s definitely more sophisticated. :wink: