typical host card is top slot while guest card is second slot. the other way to do this is a time consuming headache with grub and blacklisting and controlling boot order of devices. ( not worth it)
The problem was that the 2nd PCIe 16x slot is part of a more global IOMMU group on my motherboard so that I couldn’t isolate the GPU!
This is what I’ve done to make things work:
1st PCIe slot - NVIDIA GeForce RTX 2080 (guest GPU)
2nd PCIe slot - AMD Radeon RX 5700 XT (host GPU)
BIOS: use UEFI, not Legacy/CSM and make sure Secure Boot is disabled.
This would allow the 2nd PCIe slot to initialize during POST
Now to solve the errors with an AMD card inserted in the 2nd PCIe slot:
iommu ivhd0: Event logged [IOTLB_INV_TIMEOUT device=27.00.0 …]
BTW, no problems with any NVIDIA card
Start your favorite live USB distro (tried on Pop!_OS and Manjaro) and press “e” to edit the boot menu and add:
pci=noats