Guest has random freezing problem,ask for help

This problem has persisted since I completed PCI passthrough,but I haven’t found a solution yet.I understand that there are many reasons for freeze,but there are too many factors that make it difficult for me to start. I would like to describe my situation and hope to get some troubleshooting methods and methods to try.
Frequency of occurrence:about two seconds every few to tens of minutes
describe:

  1. When the problem occurs, the entire vm win will be completely frozen, including animations,sounds and input system.
  2. When the freeze is over, everything will return to normal, such as local music continuing to play where it was frozen.
  3. When i am playing online games,operations during freezing will not take effect,fps will not change,however, after the freeze is over, there may be some delays in order to synchronize online progress.

some try:
– I’ve tried setting up huge pages,Not getting better.

– CPU pinning:This method may be a problem with my settings, because after I finished it, the vm was stuck all the time.So I canceled it right away.

– Check libvirtd.log/win10.log…To be honest, I don’t really understand these logs.I found a few errors, but judging from the frequency of freeze, they may not be relevant to my problem.

This is my configuration for single gpu passthrough.
CPU: Ryzen 5 5600X (6 cores 12 threads,10 threads for VM | When I only allocate 8 threads for vm, it becomes laggy)

GPU: RTX 4060ti

Kernel Version:5.15.0-91-generic

OS: Ubuntu 20.04 RAM:48GB TOTAL,16GB for VM

SSD:Samsung 980Pro

grub setting:
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=pt pcie_aspm=off vfio_iommu_type1.allow_unsafe_interrupts=1 vfio_pci.disable_vga=1 vfio_pci.disable_idle_d3=1 kvm.ignore_msrs=1  vfio-pci.ids=10de:2803"

add my VM xml first.(Try the cpu pinning situation)

<domain type="kvm">
  <name>win10</name>
  <uuid>d9037881-45c4-41bd-b7b1-a32055bece63</uuid>
  <description>apex!</description>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement="static">10</vcpu>
  <cputune>
    <vcpupin vcpu="0" cpuset="1"/>
    <vcpupin vcpu="1" cpuset="7"/>
    <vcpupin vcpu="2" cpuset="2"/>
    <vcpupin vcpu="3" cpuset="8"/>
    <vcpupin vcpu="4" cpuset="3"/>
    <vcpupin vcpu="5" cpuset="9"/>
    <vcpupin vcpu="6" cpuset="4"/>
    <vcpupin vcpu="7" cpuset="10"/>
    <vcpupin vcpu="8" cpuset="5"/>
    <vcpupin vcpu="9" cpuset="11"/>
    <emulatorpin cpuset="0,6"/>
  </cputune>
  <sysinfo type="smbios">
    <bios>
      <entry name="vendor">ASUS</entry>
    </bios>
    <system>
      <entry name="manufacturer">ASUS</entry>
      <entry name="product">Windows10</entry>
      <entry name="version">21H2</entry>
    </system>
    <baseBoard>
      <entry name="manufacturer">ASUSTeK COMPUTER INC.</entry>
      <entry name="product">20BE0061MC</entry>
      <entry name="version">0B98401 Pro</entry>
      <entry name="serial">W1KS427111E</entry>
    </baseBoard>
    <chassis>
      <entry name="manufacturer">ASUSTeK COMPUTER INC.</entry>
      <entry name="version">2.12</entry>
      <entry name="serial">220299343890477</entry>
      <entry name="asset">40000101</entry>
      <entry name="sku">Type3Sku1</entry>
    </chassis>
    <oemStrings>
      <entry>myappname:some arbitrary data</entry>
      <entry>otherappname:more arbitrary data</entry>
    </oemStrings>
  </sysinfo>
  <os>
    <type arch="x86_64" machine="pc-q35-4.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/OVMF/OVMF_CODE_4M.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <smbios mode="sysinfo"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vpindex state="on"/>
      <runtime state="on"/>
      <synic state="on"/>
      <stimer state="on"/>
      <reset state="on"/>
      <vendor_id state="on" value="190236412357"/>
      <frequencies state="on"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <vmport state="off"/>
    <smm state="on"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-model" check="partial">
    <topology sockets="1" cores="5" threads="2"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="raw"/>
      <source file="/home/yisay/vm/win10.img"/>
      <target dev="sda" bus="sata"/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/yisay/Downloads/win21H2.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <boot order="2"/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:a4:e2:44"/>
      <source network="default"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x05e3"/>
        <product id="0x0608"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x0f39"/>
        <product id="0x0877"/>
      </source>
      <address type="usb" bus="0" port="4"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x8087"/>
        <product id="0x1024"/>
      </source>
      <address type="usb" bus="0" port="5"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1bcf"/>
        <product id="0x2283"/>
      </source>
      <address type="usb" bus="0" port="6"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc084"/>
      </source>
      <address type="usb" bus="0" port="7"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x08" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

and CPU INFO:

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ
  0    0      0    0 0:0:0:0           yes 3700.0000 2200.0000
  1    0      0    1 1:1:1:0           yes 3700.0000 2200.0000
  2    0      0    2 2:2:2:0           yes 3700.0000 2200.0000
  3    0      0    3 3:3:3:0           yes 3700.0000 2200.0000
  4    0      0    4 4:4:4:0           yes 3700.0000 2200.0000
  5    0      0    5 5:5:5:0           yes 3700.0000 2200.0000
  6    0      0    0 0:0:0:0           yes 3700.0000 2200.0000
  7    0      0    1 1:1:1:0           yes 3700.0000 2200.0000
  8    0      0    2 2:2:2:0           yes 3700.0000 2200.0000
  9    0      0    3 3:3:3:0           yes 3700.0000 2200.0000
  10    0      0    4 4:4:4:0           yes 3700.0000 2200.0000
  11    0      0    5 5:5:5:0           yes 3700.0000 2200.0000

check your bios for high precision event timer (hpet)…
set it to enabled if its disabled and it will stop a lot of random crashes.

if its already enabled.
then add clocksource=hpet to your grub command line and see if that helps.

Sorry for the late reply,In fact, I didn’t find the hpet settings in my bios,According to Google, ASUS has not added this item to the bios and it is disabled by default.
The output of the command cat /sys/class/rtc/rtc0/max_user_freq in ubantu is 64, which seems to be enabled.

I added clocksource=hpet to grub, but the situation remains the same,In addition, why do many opinions I looked up indicate that hpet is the culprit of lagging?

1 Like
<timer name="hpet" present="no"/>

Is the info you’re finding online saying you need HPET on the VM itself? Looks like it’s disabled in your VM config.

then the problem is likely elsewhere.

time to start looking at your logs.

sudo journalctl -b | grep error 2>/dev/null

you can grep for failed or error and send the rest to null.

| grep “search-term” 2>/dev/null

is a handy way of combing logs for errors.

System logs

System logs deal with exactly that - the Ubuntu system - as opposed to extra applications added by the user. These logs may contain information about authorizations, system daemons and system messages.

Authorization log

Location: /var/log/auth.log

Keeps track of authorization systems, such as password prompts, the sudo command and remote logins.

Daemon Log

Location: /var/log/daemon.log

Daemons are programs that run in the background, usually without user interaction. For example, display server, SSH sessions, printing services, bluetooth, and more.

Debug log

Location: /var/log/debug

Provides debugging information from the Ubuntu system and applications.

Kernel log

Location: /var/log/kern.log

Logs from the Linux kernel.

System log

Location: /var/log/syslog

Contains more information about your system. If you can’t find anything in the other logs, it’s probably here.

Application logs

Some applications also create logs in /var/log. Below are some examples.

Apache logs

Location: /var/log/apache2/ (subdirectory)

Apache creates several log files in the /var/log/apache2/ subdirectory. The access.log file records all requests made to the server to access files. error.log records all errors thrown by the server.

X11 server logs

Location: /var/log/Xorg.0.log

The X11 server creates a seperate log file for each of your displays. Display numbers start at zero, so your first display (display 0) will log to Xorg.0.log. The next display (display 1) would log to Xorg.1.log, and so on.

this seems like a lot but grepping will help.

credit to original source

https://ubuntu.com/tutorials/viewing-and-monitoring-log-files#1-overview

no,my previous attempts never even touched upon this setup,so when HEXiT mentioned it, I searched for relevant information

I helped my friend accomplish the same gpu passthrough,he has the same CPU,same ubuntu system,same host system,even the installed software is almost the same,but he doesn’t have this problem, maybe just because he uses amd GPU,or he uses the msi mainboard