Ok I somehow managed to get it working. I followed some tips from the thread below.
Basically what I did was extract the firmware files from the amdgpu-pro driver instead of just downloading them from github, used those. AND I installed Kernel 5.10, its possible xanmod kernel 5.9 does not work with RDNA2 cards yet.
amdgpu module is loaded in 5.10 but it didn’t load in 5.8 or possibly 5.9. I remember when I installed POP OS I chose NVIDIA option which probably did all sorts of things to disable amdgpu. Anyway gotta run now, everything appears to work. Will do testing later.