How to stop clock drift in VM on Threadripper 2950X?

So my problem shows up as continuous audio lag (drift) the more I use Scream+IVSHMEM for sound on the VM to the point that I get a 0.5 second of audio lag after 1 hour.

The running theory is this occurs due to the timers of the host and guest drifting apart after a while. When using QEMU audio this isn’t too big a deal because of of the small buffer size will cut things off once it overruns, but IVSHMEM is too large and it continues to drift.

Others have been able to fix the drift by using bcdedit /set useplatformclock true on the Windows 10 guest. But that hasn’t fixed the problem for me. I was wondering if anyone has been able to eliminate clock drift on guest with a system similar to mine.
CPU: AMD Threadripper 2950X
Mobo: X399 AORUS Xtreme

The following is the relevant parts of my libvirt XML

  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>8</vcpu>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='20'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='21'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='23'/>
    <emulatorpin cpuset='3,19'/>
    <iothreadpin iothread='1' cpuset='3'/>
    <memory mode='strict' nodeset='0'/>
    <type arch='x86_64' machine='pc-q35-4.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='SomeString12'/>
      <hidden state='on'/>
    <vmport state='off'/>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='2'/>
    <feature policy='require' name='topoext'/>
    <feature policy='require' name='invtsc'/>
  <clock offset='localtime'>
    <timer name='rtc' present='no' tickpolicy='catchup'/>
    <timer name='pit' present='no' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='tsc' present='yes' mode='native'/>
    <timer name='hypervclock' present='yes'/>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>

chronyd or ntpd?

I don’t care about the clock as much as the timer. The clock can be adjusted to fix the drift, but the problem with the timers constantly drifting is sending 44.1 KHz audio from guest to host ends up with more and more delay. So a constant audio steam like a game or movie will get noticeably out of sync in 30 minutes.

I couldn’t find a way of avoiding the timer drift, so instead will be updating the scream receiver to keep the audio latency from getting too large.