I suspect you are running into the reset bug on the RX580.
I too suffered the symptons you described passing through my RX580 to the guest.
I got it working by applying the following:
Isolating the GPU as described in: https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF#Isolating_the_GPU
Applying a patch to the kernel I found on the forums here:
https://forum.level1techs.com/t/attention-amd-vfio-users-do-not-update-your-bios/142685
I hope this saves you from a nasty headache 