VFIO GPU Instability

I successfully have windows 10 running in a kvm with my GPU passed through. It works fine most of the time. However, I experience game, display driver, etc. crashed when More than 3 GB of VRAM is used(as reported my windows task manager).

My system is as follows:
CPU: Intel 4790k overclocked to 4.6GHz
MOBO: Asrock Z97 Extreme4
16 GB RAM
Radeon RX 580 8GB
HP P410 RAID Controller running a raid 5 of 3 2 TB sas drives
1 500GB Samsung EVO
1 120GB OCZ vertex

I have the raid mounted as an lvm on my host(Ubuntu 18.04 with kernel 4.14.47 with the ACS override patch). Because of IOMMU groups, I therefore need the patch.

My Libvirt XML is:
win10.xml.txt (7.0 KB)

The Guest is Windows 10 Pro Version 1803
I passed through the systems USB, Sound controllers as well as the 580

To passthrough the 580, I unbind it from the system, and bind it to the vfio-pci driver using this hook:

#!/bin/bash
echo "switching 1" > /var/log/kvm-pat.log
systemctl stop gdm.service
systemctl stop display-manager.service
killall Xorg
#systemctl stop display-manager.service
sleep 1
modprobe vfio
modprobe vfio-pci

# Unbind VTconsoles
echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
  
# Unbind EFI-Framebuffer
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind


echo "1002 67df" > /sys/bus/pci/drivers/vfio-pci/new_id

echo "0000:01:00.0" > /sys/bus/pci/devices/0000:01:00.0/driver/unbind
echo "0000:01:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

echo "1002 67df" > /sys/bus/pci/drivers/vfio-pci/remove_id

echo "1002 aaf0" > /sys/bus/pci/drivers/vfio-pci/new_id

echo "0000:01:00.1" > /sys/bus/pci/devices/0000:01:00.1/driver/unbind

echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/bind

echo "1002 aaf0" > /sys/bus/pci/drivers/vfio-pci/remove_id

sleep 1

It then rebinds back to Ubuntu with:

#!/bin/bash
modprobe -r vfio-pci
modprobe -r vfio
echo 1 > /sys/bus/pci/devices/0000:01:00.0/remove
echo 1 > /sys/bus/pci/devices/0000:01:00.1/remove
echo 1 > /sys/bus/pci/rescan

# Wait 1 second to avoid possible race condition
sleep 1
  
# Re-Bind EFI-Framebuffer
#echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/bind
  
# Re-bind to virtual consoles
echo 1 > /sys/class/vtconsole/vtcon0/bind
echo 1 > tee /sys/class/vtconsole/vtcon1/bind

sleep 2

systemctl restart display-manager.service

Any ideas on what to try to fix the issue? I have done a clean reinstall of the graphics driver.