3900x, Nvidia GPU, lower performance in VM

Hello,

I may be running in circles regarding this issue but I figured I’d ask some of you fine folks for any input, this has been driving me nuts.

Setup:

  • 3900x OCed to 4.1GHz 1.2V (VID), 1.11V (SOC)
  • AsRock Taichi X570
  • 4x 16GB GSkill Trident Z CL17 @3600MHz via XMP2
  • Quadro P400 used for the Host OS ( Arch Linux ), plugged on PCIe port 3 ( bottom )
  • 2080Ti Asus Strix plugged in PCIe port 1 ( top, closest to the CPU )
  • Host SSD: ADATA_SX6000PNP on m.2 closest to the CPU
  • VM SSDs: ADATA_SP550 ( 2nd m.2 ) & Samsung_SSD_860_QVO_1TB ( SATA )
  • virtualization, IOMMU, AER Cap ( Enabled ), ACS ( Auto ) set in BIOS
  • using LVM + Luks for my Arch Linux install

The VM boots, works fine, GPU-Z as well as Nvidia Control Panel show the link speed at Gen3 x16 but I’m noticing considerably lower performance in the VM compared to on-Metal.

I’ve seen two potential solutions for this:

  • ACS Patch ( But I don’t have anything else grouped up with the 2080Ti so I don’t believe it’s required )
  • Set:
<qemu:commandline>
  <qemu:arg value="-global"/>
  <qemu:arg value="pcie-root-port.speed=8"/>
  <qemu:arg value="-global"/>
  <qemu:arg value="pcie-root-port.width=16"/>
</qemu:commandline>

With the qemu:arg try, it doesn’t work, I just get this error:

Error starting domain: internal error: qemu unexpectedly closed the monitor: 2020-05-21T15:09:54.815431Z qemu-system-x86_64: warning: This family of AMD CPU doesn't support hyperthreading(2)
Please configure -smp options properly or try enabling topoext feature.
2020-05-21T15:09:54.848928Z qemu-system-x86_64: -device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1: can't apply global pcie-root-port.speed=8: Property '.speed' not found

Not even sure what causes this issue, thoughts?

I’ve had the q35 line set to 3.1, changed it to ‘pc-q35-5.0’ after checking /usr/bin/qemu-system-x86_64 -machine help but it doesn’t seem to help with anything.

ls-iommu.sh output:

OMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 10:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12:
        00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 13:
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 14:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 15:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 16:
        01:00.0 Non-Volatile memory controller [0108]: Realtek Semiconductor Co., Ltd. Device [10ec:5763] (rev 01)
IOMMU Group 17:
        02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 18:
        03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 19:
        03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 2:
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 20:
        03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 21:
        03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
        0c:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
        0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 22:
        03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        0d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 23:
        03:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
        0e:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 24:
        04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP107GL [Quadro P400] [10de:1cb3] (rev a1)
        04:00.1 Audio device [0403]: NVIDIA Corporation GP107GL High Definition Audio Controller [10de:0fb9] (rev a1)
IOMMU Group 25:
        05:00.0 Non-Volatile memory controller [0108]: Intel Corporation SSD 600P Series [8086:f1a5] (rev 03)
IOMMU Group 26:
        06:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 27:
        07:01.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
        08:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 28:
        07:03.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 29:
        07:05.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
        0a:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 3:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 30:
        07:07.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1184e PCIe Switch Port [1b21:1184]
IOMMU Group 31:
        0f:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti Rev. A] [10de:1e07] (rev a1)
        0f:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
        0f:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Host Controller [10de:1ad6] (rev a1)
        0f:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 USB Type-C UCSI Controller [10de:1ad7] (rev a1)
IOMMU Group 32:
        10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 33:
        11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 34:
        11:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 35:
        11:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 36:
        11:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 37:
        12:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 38:
        13:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 4:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
        00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
        00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
        00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
        00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]

Looking at this I don’t think I need to apply the ACS patch

$cat /etc/sysctl.d/40-hugepage.conf

vm.nr_hugepages=8192
vm.hugetlb_shm_group=48

$cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1e07,10de:10f7,10de:1ad6,10de:1ad7

$cat /etc/modprobe.d/blacklist-nouveau.conf

blacklist nouveau
options nouveau modeset=0

$cat /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1

Checking $nvidia-smi only shows the Quadro P400.

Sorted CSM ( believe it’s set to Disabled ) so now the Quadro ( on PCI-e slot 3 ) displays POST/BIOS, the BOOT menu and LUKS decrypt prompt and so on.

Boot loader:

title Arch
linux /vmlinuz-linux
initrd /initramfs-linux.img
options cryptdevice=UUID=<DiskUUID>:volume root=/dev/mapper/volume-root quiet amd_iommu=on iommu=pt pci=noaer pci=nommconf hugepages=8192 rw vfio-pci.ids=10de:1e07,10de:10f7,10de:1ad6,10de:1ad7

$lscpu -e

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ON    MAXMHz    MINMHz
  0    0      0    0 0:0:0:0          yes 4100.0000 2200.0000
  1    0      0    1 1:1:1:0          yes 4100.0000 2200.0000
  2    0      0    2 2:2:2:0          yes 4100.0000 2200.0000
  3    0      0    3 3:3:3:1          yes 4100.0000 2200.0000
  4    0      0    4 4:4:4:1          yes 4100.0000 2200.0000
  5    0      0    5 5:5:5:1          yes 4100.0000 2200.0000
  6    0      0    6 6:6:6:2          yes 4100.0000 2200.0000
  7    0      0    7 7:7:7:2          yes 4100.0000 2200.0000
  8    0      0    8 8:8:8:2          yes 4100.0000 2200.0000
  9    0      0    9 9:9:9:3          yes 4100.0000 2200.0000
 10    0      0   10 10:10:10:3       yes 4100.0000 2200.0000
 11    0      0   11 11:11:11:3       yes 4100.0000 2200.0000
 12    0      0    0 0:0:0:0          yes 4100.0000 2200.0000
 13    0      0    1 1:1:1:0          yes 4100.0000 2200.0000
 14    0      0    2 2:2:2:0          yes 4100.0000 2200.0000
 15    0      0    3 3:3:3:1          yes 4100.0000 2200.0000
 16    0      0    4 4:4:4:1          yes 4100.0000 2200.0000
 17    0      0    5 5:5:5:1          yes 4100.0000 2200.0000
 18    0      0    6 6:6:6:2          yes 4100.0000 2200.0000
 19    0      0    7 7:7:7:2          yes 4100.0000 2200.0000
 20    0      0    8 8:8:8:2          yes 4100.0000 2200.0000
 21    0      0    9 9:9:9:3          yes 4100.0000 2200.0000
 22    0      0   10 10:10:10:3       yes 4100.0000 2200.0000
 23    0      0   11 11:11:11:3       yes 4100.0000 2200.0000

$sudo virsh vcpupin win10

VCPU   CPU Affinity
----------------------
 0      8
 1      9
 2      10
 3      11
 4      12
 5      13
 6      14
 7      15
 8      16
 9      17
 10     18
 11     19
 12     20
 13     21
 14     22
 15     23

$lstopo

$lspci -nnv

Shows all four devices on the 2080Ti with "Kernel driver in use: vfio-pci"
Last but not least, here's the XML file:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win10</name>
  <uuid>fc87d214-1499-4d4e-8a6d-bf5ac46f1c47</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <vcpu placement='static' cpuset='0-23'>16</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <vcpupin vcpu='8' cpuset='16'/>
    <vcpupin vcpu='9' cpuset='17'/>
    <vcpupin vcpu='10' cpuset='18'/>
    <vcpupin vcpu='11' cpuset='19'/>
    <vcpupin vcpu='12' cpuset='20'/>
    <vcpupin vcpu='13' cpuset='21'/>
    <vcpupin vcpu='14' cpuset='22'/>
    <vcpupin vcpu='15' cpuset='23'/>
    <emulatorpin cpuset='0-15'/>
    <iothreadpin iothread='1' cpuset='0-7'/>
    <iothreadpin iothread='2' cpuset='8-15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-5.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='host-passthrough' check='partial'>
    <topology sockets='1' cores='8' threads='2'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='disable' name='x2apic'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/ata-ADATA_SP550_2G3820048000'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/disk/by-id/ata-Samsung_SSD_860_QVO_1TB_S4CZNF0M287506M'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1c' function='0x0' multifunction='on'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:60:5f:d8'/>
      <source dev='enp10s0' mode='bridge'/>
      <model type='e1000e'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
      </source>
      <rom file='/usr/share/qemu/TU102.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='2'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <address type='usb' bus='0' port='3'/>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,hv_time,kvm=off,hv_vendor_id=null,-hypervisor'/>
    <!-- if I uncomment the below 4 lines I get the "... this family of AMD CPU doesn't support hyperthreading..." error -->
    <!-- <qemu:arg value='-global'/> -->
    <!-- <qemu:arg value='pcie-root-port.speed=8'/> -->
    <!-- <qemu:arg value='-global'/> -->
    <!-- <qemu:arg value='pcie-root-port.width=16'/> -->
  </qemu:commandline>
</domain>

Sorry for the wall of text but I figured it best to offer as much info as ( I think ) required and what I tried so far.

Curious what causes the performance drop in the VM. Seen several cases where people had almost 1:1 performance as on-metal. Any idea how I can get those qemu arguments to work?

And is the way I set my vcpu pinning correct?

I’m continuing to fiddle around with it and will come back with an update should I figure it out, maybe it’ll help anyone else having this issue.

Thanks!

Edit#1:
Noticed that after setting q35 from “pc-q35-5.0” back to 3.1, Nvidia’s Control Panel shows the PCI-e link at x1, GPU-Z still shows it at x16, didn’t notice any difference in performance though.

Updated my config a bit, added comments above changes to elaborate:

<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">
  <name>win10</name>
  <uuid>fc87d214-1499-4d4e-8a6d-bf5ac46f1c47</uuid>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <vcpu placement="static" cpuset="0-23">16</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="8"/>
    <vcpupin vcpu="1" cpuset="9"/>
    <vcpupin vcpu="2" cpuset="10"/>
    <vcpupin vcpu="3" cpuset="11"/>
    <vcpupin vcpu="4" cpuset="12"/>
    <vcpupin vcpu="5" cpuset="13"/>
    <vcpupin vcpu="6" cpuset="14"/>
    <vcpupin vcpu="7" cpuset="15"/>
    <vcpupin vcpu="8" cpuset="16"/>
    <vcpupin vcpu="9" cpuset="17"/>
    <vcpupin vcpu="10" cpuset="18"/>
    <vcpupin vcpu="11" cpuset="19"/>
    <vcpupin vcpu="12" cpuset="20"/>
    <vcpupin vcpu="13" cpuset="21"/>
    <vcpupin vcpu="14" cpuset="22"/>
    <vcpupin vcpu="15" cpuset="23"/>
    <emulatorpin cpuset="0-15"/>
    <iothreadpin iothread="1" cpuset="0-7"/>
    <iothreadpin iothread="2" cpuset="8-15"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-5.0">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/ovmf/x64/ovmf_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev="hd"/>
    <bootmenu enable="yes"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="partial">
    <topology sockets="1" cores="8" threads="2"/>
    <feature policy="require" name="topoext"/>
    <feature policy="disable" name="monitor"/>
    <feature policy="require" name="hypervisor"/>
    <feature policy="disable" name="svm"/>
    <feature policy="disable" name="x2apic"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/disk/by-id/ata-ADATA_SP550_2G3820048000"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0c" function="0x0"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="none" io="native"/>
      <source dev="/dev/disk/by-id/ata-Samsung_SSD_860_QVO_1TB_S4CZNF0M287506M"/>
      <target dev="vdb" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x0d" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x8"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x9"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0xa"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0xb"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0xc"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0xd"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0xe"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1c" function="0x0" multifunction="on"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <interface type="direct">
      <mac address="52:54:00:60:5f:d8"/>
      <source dev="enp10s0" mode="bridge"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
   <!-- Set functions to match eachother, both in <source></source> block and the one below -->
    <!-- Set all four devices belonging to the GPU on the same bus ( bus='0x08' ) as the <controller> with index="8" -->
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x0"/>
      </source>
      <rom file="/usr/share/qemu/TU102.rom"/>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0" multifunction="on"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x1"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x2"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x2"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0f" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x3"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value="-cpu"/>
    <qemu:arg value="host,hv_time,kvm=off,hv_vendor_id=null,-hypervisor"/>
  </qemu:commandline>
</domain>

Can confirm that going from q35 3.1 to 5.0 does allow the Nvidia Control-Panel to show link speed at Gen3 x16 but I’ll be damned if I actually see any difference.

Will dig deeper to see how I can sort the

"This family of AMD CPU doesn't support hyperthreading(2)"

problem when adding these lines to qemu:commandline :

<qemu:commandline>
  <qemu:arg value="-global"/>
  <qemu:arg value="pcie-root-port.speed=8"/>
  <qemu:arg value="-global"/>
  <qemu:arg value="pcie-root-port.width=16"/>
</qemu:commandline>

Hopefully I’ll be able to follow up with an update on this.

Edit#1:

Wasn’t able to sort anything out on the qemu:commandline front but I do however seem to have even less performance on the VM. Problem is I’m unsure if it’s between Windows, something the games I’m testing or virtualization. In the VM, the GPU link speed does show as Gen3 x16.
Reinstalling Windows on the VM and if that won’t help I’ll recreate the VM entirely, hope I didn’t muck up something with the pcie-root-ports in the .xml

I don’t see fake vendor name in your xml. I don’t think it’s the problem but you never know. Now when you say significant performance drops what exactly do you mean by that? Stutter, lower frame rate, freezes? Explaining problem further can lead to a a clue.
Edit: have you tried enabling MSI? I believe I had abysmal performance when ever I had emulated audio device until I did MSI on all of the devices

I mean it works, but it renders significantly less FPS, likened it to a card running on PCI-e Gen2 or Gen3 at x8. I had it set with host-passthrough, think I should just try one of the options virt-manager provides in its GUI?

Not sure what MSI is, did you mean this >MSI< ?
How would I disable/enable it in the .xml ? A qemu:args type entry I suppose?

The way I have audio is, I’ve got a powered USB Hub plugged into the Type-C port on the 2080Ti and in that I have a mouse, USB dongle and a small USB audio DAC that I use for sound.

I’m about to unplug the Quadro p400 ( PCI-e slot 3 ) to see if maybe there’s some strange thing happening, this way the only PCI-e device I’ll have plugged in will be the 2080Ti in slot 1.

Looking at what the power brick on the Hub provides and what the misc things I have plugged in I don’t see it hitting some sort of cap. But I’ll try without these plugged in.


This helps with DPC latency

In your xml config you have emulated ic9 audio device. You don’t need that unless you want to pass through audio to host. I also see spice/vnc thing… That could be causing performance drops. I don’t see emulated video device which is good and also 1 of the things that caused poor frame rate on one of my passed through cards but not the other for some reason. Basically it’s a good idea to go through virt manager window and delete all unnecessary crap that it auto created. Obviously don’t delete any PCI / SATA / USB controllers or bridges but stuff like spice, keyboard, mouse, video, serial, audio. Try to keep that list to a bare minimum.
Edit: And yes, those links above me are MSI that I was talking about. I used the one that you need to use powershell which was supper confusing since picture showed UI but what ever
Edit2: to elaborate my stutter / fps drop stuff.
Spice / video adapter caused very low frame rate.
Audio adapter caused microfreezes / stutter.
Maybe send screenshot of device manager to see what other hardware related issues you might have.

Thanks, I only tried the GUI variant so far and didn’t notice any difference ( tried booted on-metal too ). I’m reinstalling Windows on the VM and I’ll try the PS script.

I’m reinstalling the OS on the VM and I’ll try the PS script variant of the MSI fix. Tried the GUI variant ( booted on-metal too ) but didn’t seem to do much.

I’ll see with that and clean up the VM conf ( can’t remove some things from the virt-manager GUI ) and I’ll post it, hopefully with a conclusive update.

Tested some games booted Windows on-metal and there seems to be some finicky thing going on with Nvidia drivers. Some people seem to have considerable performance drops in certain games with the latest updates ( Cod: Warzone for example ). Tried that, Kingdom Come: Deliverance and Doom Eternal with various driver versions but without seeing any significant difference in performance. Will test more.

Thanks!

@robertm Would love to hear if you managed to resolve this in the meantime.

I ask because I’m having very similar issues with my own new VFIO setup (3900X + 2070 Super) where I’m seeing about 1/3 to 1/2 the FPS in VM than I am on bare metal with no obvious cause.