Some time ago I opened another thread asking for help. Unfortunately not a lot of help was provided. Now that I got this working on my setup I decided to write a guide and share it with you. The same information was also shared on reddit, FYI.
- Long post ahead -
After weeks (on and off) trying to pass my only GPU to a Windows 10/11 VM for gaming, I managed to get it working. I felt I could write a little guide to help out those wanting to do the same in Fedora (34), since most guides are written for either Ubuntu or Arch and the ones for Fedora require 2 GPUs (or are incomplete). I gathered all this information in different places, Github, Youtube, blogs, etc.
Benefits over dual boot?
- Total OS isolation.
- Preventing Windows Update from taking over your Linux partition and potentially messing things up.
- Being able to play -a few extra- online games that have an anti-cheat. We know the struggle, Proton is amazing but some games just won’t work. You’ll now be able to play Rainbow Six: Siege, Hunt Showdown, to name a few.
- VM Snapshots
- Actually quicker than having to reboot and back
- Probably great for people who record instruments over an audio interface (still haven’t tested) and miss using apps such as Ableton.
Cons (for some)
- When you shut down your VM, you’ll go back to your Linux host in a new session. It doesn’t bother me but it might bother some.
- In the long “trial and error” process to get this working, I ended up with a borked system several times. My Win10 VM would end up with a black screen which forced me to do a hard-reboot. My guess? my FS wasn’t really happy about this happening many times over short periods of time. I wasn’t able to fix a broken btrfs FS (probably due to lack of knowledge from my side) so I’d end up reinstalling everything out of laziness. From my experience XFS FSs seem to play better. Before getting it to work, I ended up with a borked XFS FS (once) which I easily fixed with xfs_repair. If this happens, don’t panic, it’s totally fixable.
My specs
- AMD Ryzen 7 3700X 8-Core Processor (16 threads)
- AMD PowerColor Red Dragon 5700XT
- Fedora 34 (GNOME)
Good to Know
- You shouldn’t pass everything to your VM. Leave some resources for your Linux host. e.g : pass 14 CPU threads out of 16 and pass 24GB RAM out of 32GB available.
- AFAIK x64 systems need a minimum of 4GB of ram (you can correct me in the comments) so if you have less than 8GB of RAM, you’re out of luck.
Pre-requisites
- Have virtualization enabled in your BIOS
- Some previous knowledge of VFIO concepts. This is NOT meant to be a full guide explaining every single detail. The guide is already longer than I expected
- Up to date system ’sudo yum update -y’
- Patience and willingness to troubleshoot. Consider this a fun mini-project
Let’s start by installing the virtualization package group.
sudo yum -y install @virtualization
or
sudo dnf -y group install Virtualization
This should install all your goodies for virtualization on KVM, including virt-manager, libvirt and qemu. Enable and start the libvirtd service.
Now let’s enable IOMMU in your system. To do this you can pass some arguments to your grub.
sudo vim /etc/default/grub
Find the line with 'GRUB_CMDLINE_LINUX=’ and add these 2:
iommu=1 amd_iommu=on
Save and exit. Now it’s time to re-build your grub by running (as sudo) 'grub2-mkconfig -o /boot/grub2/grub.cfg’ (if using BIOS) OR 'grub2-mkconfig -o /boot/efi/EFI/fedora/grub2.cfg (if using UEFI) . After our grub is recreated feel free to reboot your PC. Once you’re logged into a new session, open up a terminal and run 'cat /proc/cmdline’. If everything worked as expected, the grub parameters you passed in the previous step should be visible. In my case:
cat /proc/cmdline
BOOT_IMAGE=(hd1,gpt3)/vmlinuz-5.12.11-300.fc34.x86_64 root=/dev/mapper/fedora_localhost--live-root ro rd.lvm.lv=fedora_localhost-live/root rhgb quiet iommu=1 amd_iommu=on
VM Creation
It makes sense to create your VM now. Many guides make you configure your hooks BEFORE creating your VM, which in my opinion is a big mistake. You want to be able to create your Windows 10 VM and make sure it’s working fine before doing your magic.
Open up virt-manager. If you see an error message, it’s probable because your user has not been added to the libvirt group. To fix that:
sudo useradd -aG libvirt $USER
I won’t go into details on how to create the VM but I’ll clarify: you do NOT need to pass anything at this point. You just want to create a Windows 10 virtual machine, configure the boot options and order correctly (first the SATA device, second the Win10 ISO), select the right BIOS (Q35/UEFI OVMF_CODE.fd) and that’s pretty much it. Don’t change anything under the CPU or Memory tabs yet. You will do this after configuring the hooks.
Assuming you succeeded at creating your VM, shut it down, close virt-manager and let’s configure the libvirt hooks. In short, the libvirt hooks are scripts that will be executed at the moment you START your VM and the moment you STOP it (hence the chosen naming convention)
My directory tree for the hooks looks as follows…
[root@fedora]/etc/libvirt# pwd
/etc/libvirt
[root@fedora]/etc/libvirt# tree hooks
hooks
├── kvm.conf
├── qemu
└── qemu.d
└── win10
├── prepare
│ └── begin
│ ├── cpu_mode_performance.sh
│ ├── cpu_pinning.sh
│ └── start.sh
└── release
└── end
├── cpu_mode_ondemand.sh
├── cpu_unpinning.sh
└── stop.sh
6 directories, 8 files
… and here’s the content of each file:
- /etc/libvirt/hooks/qemu
- /etc/libvirt/hooks/kvm.conf < defining GPU IOMMU group vars
- /etc/libvirt/hooks/qemu.d/win10/prepare/begin/start.sh < self explanatory
- /etc/libvirt/hooks/qemu.d/win10/prepare/begin/cpu_mode_performance.sh < not 100% sure it’s needed cause of CPU pinning. Doesn’t hurt.
- /etc/libvirt/hooks/qemu.d/win10/prepare/begin/cpu_pinning.sh < CPU pinning
- /etc/libvirt/hooks/qemu.d/win10/release/end/stop.sh < for when shutting down your VM
- /etc/libvirt/hooks/qemu.d/win10/release/end/cpu_mode_ondemand.sh
- /etc/libvirt/hooks/qemu.d/win10/release/end/cpu_unpinning.sh < releases cores after VM is shutdown.
Needless to say, the IOMMU ids will be different from yours. Again, check this yourself with 'lspci -nnv | less’ and fix accordingly. Same goes for the CPU pinning/unpinning and start.sh/stop.sh scripts.
Having the hooks in place we can move forward to dealing with the vendor-reset bug (this is important). You can follow THIS mini guide to see how it’s done.
I followed that guide and went a bit further to make sure the kernel module was loaded at boot time. To do that, just create a file in the modules-load.d directory and echo the name of the module. In my case:
❯ pwd
/etc/modules-load.d
❯ cat vendor-reset.conf
vendor-reset
❯ dracut -fv
Reboot your PC then check if the module is loaded at boot time.
❯ lsmod | grep vendor
vendor_reset 106496 0
Now, for the last part. Open up virt-manager, click “Open” on your VM and re-configure a few things.
On the CPU tab, make sure to select host-passthrough , also make sure to configure the CPU pinning matching the number of cores/threads you defined in your cpu_pinning.sh hook. Remove everything related to Spice, tablet and tty’s. Last but definitely not least, go to Add Hardware > PCI and pass your GPU. Additionally pass any other USB device you need, in my case I also passed a bluetooth dongle (works great) , Razer Viper Ultimate (works great too) and my wireless HyperX headset. For reference I feel no delay whatsoever.
Additionally
If you have an audio interface you can also pass it as an USB device. If the audio breaks in your VM make make sure the interface is using the right Hz. Right click the Sound icon in the Windows taskbar > Sounds > Playback > Scroll down to your interface and right click > Properties > Advanced > “24 bit , 48000 Hz”. That’s how I got my Scarlett 2i2 3rd Gen interface working.
We can now press start and if we are lucky we should have a perfectly functional Windows 10 VM.
Feel free to update the OS (step 1) and also install the latest WHQL drivers (step 2) from AMD’s website. No need to use DDU, just install the WHQL on top of whatever drivers Windows Update installs.
Conclusion
I’ve been testing for 2 days and so far the performance is matching a native W10 installation. The main reason for me to do this was to be able to play Hunt: Showdown on my Linux partition, since the game uses EAC and that’s a no-go with Proton. I’m getting around 75 FPS in 3440x1440 + “recommended settings”, same as when I was playing on native Windows.
I’ll try to fix any typos I might have made, or add any information I might have missed. Hopefully this will help those running Fedora and full AMD setups!
Cheers.
