Thank you for your insight. I'm in the process of setting up my VM, but I have a few problems. Could you maybe help me out a bit?
I'm no expert in GPU passthrough(this is the first time I've done it), but I've found the process to be straightforward and not much different than setting up a normal qemu/KVM instance, all I've done is follow one of the guides about setting up the primary GPU passthrough with OVMF(mainly because I only have 1 GPU).
Is there a specific part of the process you need help with?
I currently have two problems. I mostly followed this guide and in step 7c I had to add something to /etc/libvirt/qemu.conf. It works but I get much longer boot times after adding it. My second problem is that my vm won't boot from a windows ISO. When I start it, I get a UEFI console and if I start the BOOTX64.efi file from there nothing happens. I tried two different ISOs and both didn't work.
1 Like
Can you describe your hardware/OS for me? Also, please paste the output of the following scripts so I can have a look at iommu groups:
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done;I have a Ryzen 7 1700 on an X370 Taichi running Fedora 25 with Kernel 4.10.17-200. The 980 Ti is successfully claimed by the vfio-pci driver and I can pass it to the vm. I think my main problem is my libvirt xml file, but I'm not sure what exactly I did wrong. The IOMMU groups are the following:
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 11 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
IOMMU Group 11 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
IOMMU Group 11 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
IOMMU Group 11 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 11 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 09:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 11 0b:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 11 0c:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU Group 11 0d:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788]
IOMMU Group 12 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
IOMMU Group 12 0f:00.1 Audio device [0403]: NVIDIA Corporation GM200 High Definition Audio [10de:0fb0] (rev a1)
IOMMU Group 13 10:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [10de:1d01] (rev a1)
IOMMU Group 13 10:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb8] (rev a1)
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 7 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 7 11:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 7 11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]
IOMMU Group 8 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 8 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 8 12:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 8 12:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]
IOMMU Group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)As you see my graphics cards are in separate IOMMU groups.
I use a simple shell script that invokes qemu instead of libvirt(not a big fan of XML), maybe you can give it a shot:
#!/bin/bash -e
socket="/run/qemu/win8.sock"
veth="vb-win8"
bridge="br0"
setup() {
mkdir -p /run/qemu
}
quit() {
echo system_powerdown | nc -U $socket
}
veth_setup() {
ip tuntap add dev $veth mode tap
ip link set $veth up
brctl addif $bridge $veth
}
veth_teardown() {
ip tuntap del dev $veth mode tap
}
run_qemu() {
exec qemu-system-x86_64 \
-bios /virt/win8/OVMF.fd \
-enable-kvm \
-cpu Penryn,kvm=off,hv_vendor_id=HyperKVM,hv_relaxed,hv_time,hv_spinlocks=0x1fff \
-smp cores=4,sockets=1,threads=1 \
-m 8G \
-drive file=/virt/win8/disk.qcow2,if=virtio,cache=unsafe,index=1 \
-boot d \
-drive file=/virt/win8/virtio-win-0.1.126.iso,media=cdrom,index=2 \
-drive file=/virt/win8/w8eng.iso,media=cdrom,index=3 \
-net nic,model=virtio -net tap,ifname=$veth,script=no,downscript=no \
-rtc base=localtime,clock=host \
-device vfio-pci,host=29:00.0,multifunction=on,romfile=/virt/win8/Asus.RX460.2048.160817.rom \
-device vfio-pci,host=29:00.1 \
-nographic -vga none \
-usb -usbdevice host:046d:c52b \
-device usb-kbd -device usb-mouse \
-monitor unix:$socket,server,nowait
}
setup
veth_setup
(run_qemu) &
trap "veth_teardown" EXIT ERR
trap "quit" TERM
waitThis script assumes a bunch of files are in /virt/win8, so you need to adapt for your case(also fix any PCI/USB ids that are passed to the VM, usb is the mouse/keyboard I use). Note that the -boot d(and the two lines below it) are used for mounting windows 8 iso and virtio drivers iso, and is only required during installation.
Also note that it assumes a bridge br0 is connected to the host. I think libvirt by default creates a bridge virbr0 so you may need to adapt network setup.
If this script works for you, you can also create a systemd unit that takes care of automatically starting/stopping the VM(assuming the above script is in /usr/local/bin/win8-qemu):
[Unit]
Description=Windows 8 gaming VM
After=network.target
[Service]
Type=simple
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=25
ExecStart=/usr/local/bin/win8-qemu
[Install]
WantedBy=multi-user.target
1 Like
Thanks, it look promising. I'll try to adapt it to my system. Could you explain to me why you virtualize a Penryn CPU instead of host-passthrough?
When I first started using npt=0, I noticed that qemu only allocated 1 CPU for the guest even though I asked it to allocate 4. Using -cpu Penryn fixed that. Also, using passmark with -cpu host results in BSOD.
Another important thing to notice in my script, the line -drive file=/virt/win8/disk.qcow2,if=virtio,cache=unsafe,index=1 \` effectively sends guest disk writes to memory with periodic flushes to disk. This improves write performance by a lot but makes it possible for you to lose data and even corrupt the guest filesystem. The reason I use that is because I have a good UPS that won't let my system to halt due to lack of power, but you may want to removecache=unsafeor addcache=none`(though it is best to leave the default by not specifying a cache)
Does using another cpu model result in less performance?
I currently have the problem that I get a bunch of errors if I don't use the romfile option and if I use a romfile that I dumped with nvflash on windows, the gpu doesn't work at all in the vm. I tried to dump the rom on linux like this:
cd /sys/bus/pci/devices/0000:01:00.0/
echo 1 > rom
cat rom > /usr/share/kvm/vbios.bin
echo 0 > rom
but I get "Input/output error".
Does using another cpu model result in less performance?
Probably some CPU features won't be available for the guest, but I don't think this makes significant difference.
I tried to dump the rom on linux like this:
I don't think you can dump the ROM from linux, I had to use GPU-z on windows(dualboot)
An easier option is to download from here: https://www.techpowerup.com/vgabios/
I tried a bios I extraced with GPU-Z and it also doesn't work.
It seems to work now if I remove the romfile option and add the kernel parameter video=efifb:off
I have tested host-passthrough and the Penryn cpu model with 4 cores in GTA 5 against a bare metal Windows installation with 8 and 4 cores. I get 30-40 FPS in the VM and 90-120 FPS on bare metal. Turning NPT on or off doesn't change much, so the CPU performance in the VM is a real issue for me.
It is really strange that you get such a high performance hit from CPU alone. Can you confirm that npt is off by checking the output of cat /sys/module/kvm_amd/parameters/npt ?
I just noticed that I forgot to regenerate my initramfs after I changed it to npt=0. I'll test it again later.
I tested it with npt off now and the FPS are a little better but they are fluctuating a lot which makes it look very choppy.
1 Like
Something you can try is to pin the VM to 4 CPU cores(either the first 4 or last 4). Some people reported increased FPS and fixed stuttering when doing it. I can't say much because I'm not a heavy gamer and only tried light titles(the heaviest game I've played is GTA IV which works ok for me)
Essentially ryzen is the same as 2x 4 core CPU in one socket, the first 4 cores communicate with the last 4 via infinity fabric which has performance linked to memory clock speed. This is why benchmarking people experience FPS increase with high clocked memory on ryzen, and also why some games started being showing better FPS after being "optimized" for ryzen.
With qemu/kvm the same we have the same problem, splitting VM processor time across all 8 cores is doomed to have a lot of cache misses(even if you only assign 2 or 4 cores to the VM, the time will be split across all cores). Using the systemd script above, you can add the following configuration to PIN the last 4 cores to the VM:
[Unit]
Description=Windows 8 gaming VM
After=network.target
[Service]
CPUAffinity=8 9 10 11 12 13 14 15
Type=simple
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=25
ExecStart=/usr/local/bin/win8-qemu
[Install]
WantedBy=multi-user.targetif using libvirt, use a different method of setting affinity(google and shouldn't be hard to find)
I spent the last 3 days debugging this and found a solution to the lack of performance with ntp enabled. See this thread for details:
https://lists.linuxfoundation.org/pipermail/iommu/2017-October/024823.html
2 Likes
This is your work?
Thank you!
Yes, and you’re welcome 
There is an official patch here: https://patchwork.kernel.org/patch/10027525/
This will likely be included upstream in the kernel in the very near future.
2 Likes