Nvidia VFIO with driver 545

I had a pretty nice VFIO setup going on AM5 with an RTX 3060. I would bind the GPU to vfio on boot, and after starting my DE could reclaim the GPU and use it for compute/offloading graphics with prime; or bind/unbind to VMs as needed.

But since updating to the nvidia driver 545, when rebinding the GPU to host, my DE (gnome on Wayland) will freeze (where I can save it by switching in and out of the tty) or more likely crash. After this gnome will leech on to the GPU making it so that I can’t unbind it anymore without stopping the DE (in which case I may as well reboot since my stuff won’t stay open). Before, on driver 539 I could reclaim the GPU, do stuff on linux, then bind it to a VM again with no issues (sometimes I’d need to kill firefox since it would sometimes leech on to the GPU but no bigger issues).

The system is:

Ryzen 7950x (iGPU is primary in Bios)
proart x670e
Fedora 39 with Nvidia drivers from rpmfusion.

Anyone having the same issue or knows a workaround?