High DPC latency + Audio Crackle

Software & Operating Systems VFIO
,
1

Hello I am new to VFIO but not new to Linux. First I will post Specs then Configs/Logs

Things I’ve tried:
Pinned CPUs cores 2-7
Used isolcpus
Recompiled Manjaro Kernel with VFIO patches
Installed VirtIO drivers for all devices and Device Manager
Used MSI v2 tool to enable MSI for GPU and GPU Audio

OS:Manjaro
Kernel: Personally Compiled 5.6.11-1 with VFIO Patches.

Specs:
Gigabyte Z390 AORUS Motherboard
i7-9700KF 5.0GHz @1.35V (NOT multithreaded)
EVGA 2080 Super XC HYBRID (Passed to VM)
GT 210 ( Arch Linux Host)
EVGA 650W Gold+ PSU

VM XML:

<domain type="kvm">
  <name>win10</name>
  <uuid>26314a1c-8d6c-4639-b296-2dde5ef4377b</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">12386304</memory>
  <currentMemory unit="KiB">12386304</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static" current="6">36</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/>
    <vcpupin vcpu="1" cpuset="5"/>
    <vcpupin vcpu="2" cpuset="2"/>
    <vcpupin vcpu="3" cpuset="6"/>
    <vcpupin vcpu="4" cpuset="3"/>
    <vcpupin vcpu="5" cpuset="7"/>
    <emulatorpin cpuset="0,4"/>
  </cputune>
  <os>
    <type arch="x86_64" machine="pc-q35-4.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <bootmenu enable="yes"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="anything"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none">
    <topology sockets="1" cores="6" threads="6"/>
    <feature policy="require" name="invtsc"/>
    <feature policy="require" name="topoext"/>
  </cpu>
  <clock offset="utc">
    <timer name="tsc" present="yes" mode="native"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/ech0/Downloads/Microsoft_Windows_10_Enterprise_Version_1903_X64-CYGiSO/cyg-w1903x64en.iso"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/ech0/Downloads/virtio-win-0.1.171.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/win10.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <boot order="2"/>
      <address type="pci" domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-to-pci-bridge">
      <model name="pcie-pci-bridge"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <controller type="scsi" index="0" model="virtio-scsi">
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </controller>
    <controller type="scsi" index="1" model="virtio-scsi">
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:bd:69:df"/>
      <source network="br1"/>
      <model type="virtio"/>
      <link state="up"/>
      <address type="pci" domain="0x0000" bus="0x09" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
      <gl enable="no"/>
    </graphics>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc539"/>
      </source>
      <address type="usb" bus="0" port="2"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x0b05"/>
        <product id="0x184d"/>
      </source>
      <address type="usb" bus="0" port="3"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1852"/>
        <product id="0x7022"/>
      </source>
      <address type="usb" bus="0" port="4"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

Grub Boot Line
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt isolcpus=2-7 nohz_full=2-7 rcu_nocbs=2-7 transparent_hugepage=never rd.driver.pre=vfio-pci vfio-pci.ids=10de:1e81,10de:10f8 pcie_acs_override=downstream,mul>

Passed Through Devices:
2080 Super
Nvidia HD Audio Controller (GPU Audio)
Logitech Wireless Pro
YROCOM C&C Co., LTD DigiHug USB Audio (Fii) Olypus 2 AMP + HD6xx headphones)

I’ve tried Plugging the USB DAC for my headset into the dedicated USB-DAC port on my Mobo as well as a standard USB-3.0 port

2

Ok still doing some testing but I think passing the onboard XHCI 3.1 USB Controller to the VM fixed audio issues. Not ideal but I guess it’s that or getting a separate soundcard to take up my last PCI Slot.

Or I could set this up Scream Virtual Audio Card/Server as a last resort but I’d rather not

Still would like some advice on improving the High DPC Latency tho!

3

If you experience high dpc latency issues on windows,
then it could be a driver related issue.
You could use a tool like latency mon, that might give you some clue’s,
where to start looking.

I assume you run the latest bios already?

4

I actually had already used Latencymon as that is how I fouund out about this issues.

I think it hinted at the nvidia driver or the Nvidia Audio driver (even tho I am using a USB3 AMP+ Headset for audio)

I am not on the latest BIOS actually. I didn’t bother updating to the latest as I thought the only reason to update was to fix IOMMMU groupings but I read that IOMMU grouping was screwed on this Board anyways so I didn’t bother.

After I passed my XHCI 3.1 controller to the VM I played COD MW Moshpit online and the SIngleplayer Compaign and has zero issues. Just annoyed with the values in LatancyMod

5

Still really high DPC Latency. I have modified my grubline a bit
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on iommu=pt isolcpus=2-7 nohz_full=2-7 rcu_nocbs=2-7 kvm.ignore_msrs=1 transparent_hugepage=never rd.driver.pre=vfio-pci vfio-pci.ids=10de:1e81,10de:10f8 pcie_acs_override=downstream,multifunction apparmor=1 security=apparmor udev.log_priority=3>"

No audio issues since passing through USB Controller

Can someone check my XML to make sure I am passing my Cores correctly

My Numa layout
https://justpaste.it/46ssq

EDIT 1: OK I played around with the CPU part of the XML and I am getting higher R20 CPU Scores now. Got 3700 for MultiCore test for my [email protected].

That seems to be about what i should get. Yet my FPS in games seems to be down aboout 25% or so.

For Example in BFV at Ultra Settings (No RayTracing): 80-110FPS

<vcpu placement="static" current="8">16</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu="0" cpuset="2"/>
    <vcpupin vcpu="1" cpuset="3"/>
    <vcpupin vcpu="2" cpuset="4"/>
    <vcpupin vcpu="3" cpuset="5"/>
    <vcpupin vcpu="4" cpuset="6"/>
    <vcpupin vcpu="5" cpuset="7"/>
    <emulatorpin cpuset="0-1"/>
  </cputune>