See this tutorial and check the Nvidia specific part. You need to edit the xml configuration.
I personally don’t use virt-manager but a plain and simple bash script with the qemu command - much easier to debug and edit.