Navi GPU Passthrough Freezes Frequently

Hi all! I have been trying to get single GPU passthrough working on my Gentoo system for the past couple of days. I use a 5700xt and a Xeon E5-1680v2. Setting up the VM (no GPU) went without a hitch, and had zero issues. At first, I wanted to be able to switch between the host and client without a reboot, but soon discovered that the Navi reset bug is an impassable barrier for now (pink boot window and code 43). As such, I instead opted to create a separate efi entry with amdgpu masked and various kvm options enabled, which would start the VM at boot. Currently I use ssh to start the VM, but will set up a script once it is working. Libvirt autobinds the GPU to the VM, and it boots. On initial startup, everything works well. The graphics driver even installs successfully, and no code 43 errors. The VM can reboot as well (I have the navi reset patch installed). However, once the graphics drivers are installed, the VM will occasionally lock up for a minute or two before resuming. There is no black screen, no flashing, just applications get confused because they have been frozen for so long. Although it happens at random times as well, it can be triggered almost immediately by running a game/anything heavily utilizing the GPU. The game just runs correctly for a few seconds and then freezes. Through SSH, I can tell that it has not hard locked the host, but QEMU does max a few cores until it goes away. dmesg doesn’t complain either. I have tried a number of things to fix this. Switching graphics driver versions, enabling MSI, disabling MSRs, switching to QEMU master, all to no avail. I allocate 12/16 cores to the VM, and 64/128GB of RAM. When the freezes happen, neither the host nor guest are low on RAM, and the host is consuming almost zero CPU. Any ideas? Any additional configuration details I should share? Thanks in advance!