Passthrough AMD GPU: driver timeout into reboot with error 43

Hello, I am having some issues with GPU passthrough.

System is Ubuntu 20.10 (updated from 20.04 thinking it might fix my issue)
Kernel 5.9.0 (was using the vega fix in kernel until I found the vendor-reset module, when I removed the vega fix and is just custom built, no changes from git master)
Motherboard: ROG STRIX X470-F GAMING
GPU: Sapphire 5700XT
GPU for passthrough: RX Vega 64

lspci -nn command:
0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c1)
0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]

grub command line:
GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash amd_iommu=on iommu=pt vfio-pci.ids=1002:687f,1002:aaf8”

My problem is the following:
The Windows VM boots fine however, after some time (random, from seconds to hours at one point), I get artifacts on screen and the AMD driver crashes saying that it detected a timeout. After several of these driver crashes, the VM resets and the GPU is in error 43. (spice video is working at this point)
I am using Video card driver virtualisation detection bypass, as well as hiding the KVM CPU leaf (probably not needed as it is not NVIDIA)
A host reboot will make the VM boot with the GPU once more.

I am using the lastest AMD GPU driver in Windows: Adrenalin 2020 Edition 20.11.2 Recommended (WHQL)
My libvirt xml file: libvirt.xml.txt (9.2 KB)