I was originally planning to use my RX 580 for GPU passthrough because AMD doesn’t try to prevent the use of consumer-grade Radeons in VMs. However, I got bitten by the “Radeon reset bug” and couldn’t get my VM to a stable state with AMD.
Basically, this bug prevents you from rebooting your VM. Instead, you have to reboot the VM and the host OS. It’s very annoying.
So now my setup consists of an RX 580 running on a Fedora Core 28 host, and a GTX 1050 Ti attached to a Windows 10 VM. The problem with Nvidia in a VM is that it takes extra trickery to fool the Nvidia driver to load in a VM. Forum member GrayBoltWolf has an excellent tutorial here The guide pertains to Debian Stretch, but the Nvidia specific configuration should be applicable to any KVM install.