Hello,
I wanted to ugprade to Ryzen since I heard about it because I wanted to switch out my CPU to get IOMMU support. I was holding out because I heard of the issue with IOMMU grouping. Since that seems to have improved with Agesa 1.0.0.6 I ordered a R7 1700 and a X370 Taichi. I flashed a beta BIOS with the new microcode on it and enabled the ACS option in the bios. Now the IOMMU groups seem fine, because the devices in the cpu slots are in separate IOMMU groups.
Now I'm getting second thoughts because I heard of issues inside the VM with NPT turned on or off. Do you think that this issue could also be resolved with software updates? If not, I have to think about return my new hardware because that's the main reason I bought it
There's indeed a problem with kvm npt, but it is not a blocker for gpu passthrough. Here's what I learned from testing my new R7 1700 system:
With npt=1, you get very good KVM CPU performance, possibly very near native speed, but on some workflows the GPU seems to have bad performance. Note that this is not for all applications/games, and depending on what software you run there may not even be a difference. For example, I have a radeon rx460(a low end card) and with npt=1 I got the same 1920x1080(AA x4) score on unigine heaven benchmark as running on bare metal(I keep a windows partition to perform these comparisons as ryzen-supporting software improves). I also tried some old games such as neverwinter nights 2 and the performance doesn't seem to suffer.
Another strange behavior: on windows 10 as guest, I got bad performance on google chrome when gpu acceleration is enabled, but on windows 8 google chrome runs perfectly! This strange behavior makes me think this npt=1 is a pure software issue that may soon be fixed in the future. Maybe @wendell learned something about this from his AMD contacts
Now, there seems to be some wrong ideas that running kvm with npt=0 yields to horrible CPU performance(while fixing GPU performance). From my experience this is definitely not the case! What seems to happen is that when npt=0 and -cpu host
is passed as qemu parameter(apparently the default for libvirt), qemu will only allocate 1 CPU for the guest, so this could give the impression of the bad CPU performance. If you tell QEMU to emulate an older CPU, it will pass the requested number of cores to the VM which will yield a very acceptable performance. The CPU performance is not as good as with npt=1, but more than good enough for gaming: Currently using a VM as headless gaming server with in home streaming and it works very well(can get consistent 40-60 fps on GTA IV, but using a RX 460)!
Thank you for your insight. I'm in the process of setting up my VM, but I have a few problems. Could you maybe help me out a bit?
I'm no expert in GPU passthrough(this is the first time I've done it), but I've found the process to be straightforward and not much different than setting up a normal qemu/KVM instance, all I've done is follow one of the guides about setting up the primary GPU passthrough with OVMF(mainly because I only have 1 GPU).
Is there a specific part of the process you need help with?
I currently have two problems. I mostly followed this guide and in step 7c I had to add something to /etc/libvirt/qemu.conf. It works but I get much longer boot times after adding it. My second problem is that my vm won't boot from a windows ISO. When I start it, I get a UEFI console and if I start the BOOTX64.efi file from there nothing happens. I tried two different ISOs and both didn't work.
Can you describe your hardware/OS for me? Also, please paste the output of the following scripts so I can have a look at iommu groups:
#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
n=${d#*/iommu_groups/*}; n=${n%%/*}
printf 'IOMMU Group %s ' "$n"
lspci -nns "${d##*/}"
done;
I have a Ryzen 7 1700 on an X370 Taichi running Fedora 25 with Kernel 4.10.17-200. The 980 Ti is successfully claimed by the vfio-pci driver and I can pass it to the vm. I think my main problem is my libvirt xml file, but I'm not sure what exactly I did wrong. The IOMMU groups are the following:
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 10 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1460]
IOMMU Group 10 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1461]
IOMMU Group 10 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1462]
IOMMU Group 10 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1463]
IOMMU Group 10 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1464]
IOMMU Group 10 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1465]
IOMMU Group 10 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1466]
IOMMU Group 10 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1467]
IOMMU Group 11 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b9] (rev 02)
IOMMU Group 11 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b5] (rev 02)
IOMMU Group 11 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b0] (rev 02)
IOMMU Group 11 04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 04:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43b4] (rev 02)
IOMMU Group 11 06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 02)
IOMMU Group 11 07:00.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:01.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:03.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:05.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 08:07.0 PCI bridge [0604]: ASMedia Technology Inc. Device [1b21:1184]
IOMMU Group 11 09:00.0 Network controller [0280]: Intel Corporation Device [8086:24fb] (rev 10)
IOMMU Group 11 0b:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 11 0c:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 04)
IOMMU Group 11 0d:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788]
IOMMU Group 12 0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX 980 Ti] [10de:17c8] (rev a1)
IOMMU Group 12 0f:00.1 Audio device [0403]: NVIDIA Corporation GM200 High Definition Audio [10de:0fb0] (rev a1)
IOMMU Group 13 10:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP108 [10de:1d01] (rev a1)
IOMMU Group 13 10:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fb8] (rev a1)
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 5 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 7 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 7 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 7 11:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 7 11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:145c]
IOMMU Group 8 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1452]
IOMMU Group 8 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1454]
IOMMU Group 8 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 8 12:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 8 12:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]
IOMMU Group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 9 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
As you see my graphics cards are in separate IOMMU groups.
I use a simple shell script that invokes qemu instead of libvirt(not a big fan of XML), maybe you can give it a shot:
#!/bin/bash -e
socket="/run/qemu/win8.sock"
veth="vb-win8"
bridge="br0"
setup() {
mkdir -p /run/qemu
}
quit() {
echo system_powerdown | nc -U $socket
}
veth_setup() {
ip tuntap add dev $veth mode tap
ip link set $veth up
brctl addif $bridge $veth
}
veth_teardown() {
ip tuntap del dev $veth mode tap
}
run_qemu() {
exec qemu-system-x86_64 \
-bios /virt/win8/OVMF.fd \
-enable-kvm \
-cpu Penryn,kvm=off,hv_vendor_id=HyperKVM,hv_relaxed,hv_time,hv_spinlocks=0x1fff \
-smp cores=4,sockets=1,threads=1 \
-m 8G \
-drive file=/virt/win8/disk.qcow2,if=virtio,cache=unsafe,index=1 \
-boot d \
-drive file=/virt/win8/virtio-win-0.1.126.iso,media=cdrom,index=2 \
-drive file=/virt/win8/w8eng.iso,media=cdrom,index=3 \
-net nic,model=virtio -net tap,ifname=$veth,script=no,downscript=no \
-rtc base=localtime,clock=host \
-device vfio-pci,host=29:00.0,multifunction=on,romfile=/virt/win8/Asus.RX460.2048.160817.rom \
-device vfio-pci,host=29:00.1 \
-nographic -vga none \
-usb -usbdevice host:046d:c52b \
-device usb-kbd -device usb-mouse \
-monitor unix:$socket,server,nowait
}
setup
veth_setup
(run_qemu) &
trap "veth_teardown" EXIT ERR
trap "quit" TERM
wait
This script assumes a bunch of files are in /virt/win8
, so you need to adapt for your case(also fix any PCI/USB ids that are passed to the VM, usb is the mouse/keyboard I use). Note that the -boot d
(and the two lines below it) are used for mounting windows 8 iso and virtio drivers iso, and is only required during installation.
Also note that it assumes a bridge br0
is connected to the host. I think libvirt by default creates a bridge virbr0 so you may need to adapt network setup.
If this script works for you, you can also create a systemd unit that takes care of automatically starting/stopping the VM(assuming the above script is in /usr/local/bin/win8-qemu):
[Unit]
Description=Windows 8 gaming VM
After=network.target
[Service]
Type=simple
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=25
ExecStart=/usr/local/bin/win8-qemu
[Install]
WantedBy=multi-user.target
Thanks, it look promising. I'll try to adapt it to my system. Could you explain to me why you virtualize a Penryn CPU instead of host-passthrough?
When I first started using npt=0, I noticed that qemu only allocated 1 CPU for the guest even though I asked it to allocate 4. Using -cpu Penryn
fixed that. Also, using passmark with -cpu host
results in BSOD.
Another important thing to notice in my script, the line -drive file=/virt/win8/disk.qcow2,if=virtio,cache=unsafe,index=1 \` effectively sends guest disk writes to memory with periodic flushes to disk. This improves write performance by a lot but makes it possible for you to lose data and even corrupt the guest filesystem. The reason I use that is because I have a good UPS that won't let my system to halt due to lack of power, but you may want to remove
cache=unsafeor add
cache=none`(though it is best to leave the default by not specifying a cache)
Does using another cpu model result in less performance?
I currently have the problem that I get a bunch of errors if I don't use the romfile option and if I use a romfile that I dumped with nvflash on windows, the gpu doesn't work at all in the vm. I tried to dump the rom on linux like this:
cd /sys/bus/pci/devices/0000:01:00.0/
echo 1 > rom
cat rom > /usr/share/kvm/vbios.bin
echo 0 > rom
but I get "Input/output error".
Does using another cpu model result in less performance?
Probably some CPU features won't be available for the guest, but I don't think this makes significant difference.
I tried to dump the rom on linux like this:
I don't think you can dump the ROM from linux, I had to use GPU-z on windows(dualboot)
An easier option is to download from here: https://www.techpowerup.com/vgabios/
I tried a bios I extraced with GPU-Z and it also doesn't work.
It seems to work now if I remove the romfile option and add the kernel parameter video=efifb:off
I have tested host-passthrough and the Penryn cpu model with 4 cores in GTA 5 against a bare metal Windows installation with 8 and 4 cores. I get 30-40 FPS in the VM and 90-120 FPS on bare metal. Turning NPT on or off doesn't change much, so the CPU performance in the VM is a real issue for me.
It is really strange that you get such a high performance hit from CPU alone. Can you confirm that npt is off by checking the output of cat /sys/module/kvm_amd/parameters/npt
?
I just noticed that I forgot to regenerate my initramfs after I changed it to npt=0. I'll test it again later.
I tested it with npt off now and the FPS are a little better but they are fluctuating a lot which makes it look very choppy.
Something you can try is to pin the VM to 4 CPU cores(either the first 4 or last 4). Some people reported increased FPS and fixed stuttering when doing it. I can't say much because I'm not a heavy gamer and only tried light titles(the heaviest game I've played is GTA IV which works ok for me)
Essentially ryzen is the same as 2x 4 core CPU in one socket, the first 4 cores communicate with the last 4 via infinity fabric which has performance linked to memory clock speed. This is why benchmarking people experience FPS increase with high clocked memory on ryzen, and also why some games started being showing better FPS after being "optimized" for ryzen.
With qemu/kvm the same we have the same problem, splitting VM processor time across all 8 cores is doomed to have a lot of cache misses(even if you only assign 2 or 4 cores to the VM, the time will be split across all cores). Using the systemd script above, you can add the following configuration to PIN the last 4 cores to the VM:
[Unit]
Description=Windows 8 gaming VM
After=network.target
[Service]
CPUAffinity=8 9 10 11 12 13 14 15
Type=simple
KillMode=process
KillSignal=SIGTERM
TimeoutStopSec=25
ExecStart=/usr/local/bin/win8-qemu
[Install]
WantedBy=multi-user.target
if using libvirt, use a different method of setting affinity(google and shouldn't be hard to find)
I spent the last 3 days debugging this and found a solution to the lack of performance with ntp enabled. See this thread for details:
https://lists.linuxfoundation.org/pipermail/iommu/2017-October/024823.html