My reference computer (the one I actually got passthrough working on) is an Epyc 7272/ROMED8-2T with a Radeon HD6670 passed through to a Win10 Guest. Debian 10.6 Host. (And it was easy and painless!)
On the reference machine, here’s what I have:
/etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX=""
Everything is commented out in /etc/initramfs-tools/modules.
In /etc/modules, it doesn’t look like anything graphics card related:
vhost_net
# Generated by sensors-detect on Thu Dec 31 03:10:17 2020
# Chip drivers
nct6775
# Generated by sensors-detect on Thu Dec 31 03:11:17 2020
# Chip drivers
nct6775
In /etc/modprobe.d/vfio.conf:
options vfio-pci ids=1002:6758,1002:aa90
In /etc/modules-load.d/vfio-pci.conf:
vfio-pci
So I’m not sure what’s going on with the Ryzen machine. But I’m using the known-good Epyc machine to narrow-down what actually needs to be set to eliminate variables.
On my Epyc test machine, when Spice is running in virt-manager, the passed-through card and Spice act like a dual monitor system. I can drag windows between them. I had to make the passthrough primary in wodows though, and now Spice acts like a second monitor. Before that, the passthrough was the secondary monitor with the wallpaper but not the start menu.
On this Ryzen machine, the passed card screen is always black and the monitor goes to sleep reading no input. (So it’s not outputting black video, it’s outputting nothing).
Does anyone know if detecting Spice is part of Nvidia’s VM detection algorithm?
On the Epyc machine (AMD chip), it seems to be working without any options in /etc/default/grub/. I reran update-initramfs and update-grub and rebooted to be sure, and the passthrough card still works.
grep -rn '/etc/grub.d/' -e 'iommu' produces no output, so it doesn’t seem to be set in any of the grub config files. But dmesg | grep iommu shows output, so it’s still loading somehow.
I had to make two changes to get the XML to validate; there were no closing single quotes on two of the lines and no self-closing tag slashes on those lines. (The quote above is edited to reflect those changes.) But after that, trying to start the machine, I get:
error: Failed to start domain win10
error: internal error: qemu unexpectedly closed the monitor: 2021-01-09T07:26:44.769105Z qemu-system-x86_64: Property '.hy' not found
If I remove the ,-hy, I get:
error: Failed to start domain win10
error: internal error: qemu unexpectedly closed the monitor: audio: warning: Using timer based audio emulation
If I remove the last line, <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>, I get:
error: Failed to start domain win10
error: internal error: process exited while connecting to monitor: 2021-01-09T07:30:37.880715Z qemu-system-x86_64: -device ioh3420,bus=pci.0,addr=1b.0,multifunction=on,p: Bus 'pci.0' not found
If I remove <qemu:arg value='ioh3420,bus=pci.0,addr=1b.0,multifunction=on,p'>, I get:
error: internal error: process exited while connecting to monitor: 2021-01-09T07:31:58.349417Z qemu-system-x86_64: host,hv_time,kvm=off,hv_vendor_id=whatever: Could not open 'host,hv_time,kvm=off,hv_vendor_id=whatever': No such file or directory
Every time I removed a line, I got a different error, but once all of the <qemu:commandline> stuff was removed, the machine did start. Still no output on the PCIe card though.
The top line of my XML file reads:
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
…because I think you need to include the qemu xml namespace to make the qemu: elements work.