I have spent the last 40 hours or so rebooting my computer hundreds of times trying to figure this out, but it is kicking my ass. I am no stranger to VFIO and passthru. I been using Intel iGPU for Linux host and passing through Nvidia to Windows guest with no problems. Until now, where I am trying to pass through two Nvidias to the same guest. My motherboard has a x8/x8 setup where both cards are in the same IOMMU group. This is fine because I want to use both cards on the same VM. I cannot get graphical output on the iGPU after Linux and the initrd takes over from the bootloader. The main culprit is switching between onboard/PCIe as the primary graphics. When set to PCIe I can load up using either nouveau or nvidia drivers with both video cards in the system. I can even turn on iGPU multi monitor and have a screen hooked to the iGPU/motherboard and it works (as long as PCIe is still the ‘primary’). Once I set iGPU/onboard as primary I can no longer get graphical output from anything unless I remove my 2nd Nvidia card. My main install is arch which I have VFIO stub working just fine and it grabs all my PCIe devices as it should (except the 2nd Nvidia). I have the device IDs for both Nvidia cards provided. I have the vfio modules added to my initrd via the mkinitcpio modules hook. With the Manjaro liveusb I have tried a million different combinations of kernel parameters and free/nonfree driver option. I’ve tried blacklisting nouveau/nvidia, I’ve tried all the different combinations of nomodeset/modesets. I’ve even tried pci=noacpi, noapic, acpi=off.
TL;DR: My VFIO setup with Intel iGPU as Linux host and Nvidia as guest passthru works just fine, and without changing any configs, if I add my 2nd Nvidia card I can’t get iGPU graphical output on boot. It just hangs on a blank screen (there is a signal going to monitor, I know because backlight is on and the monitor doesn’t go to sleep) I cannot switch TTY or anything here.
The Nvidia cards I have are 1660 Ti & 1660 SUPER. I have both of their PCIe device IDs listed in my vfio-pci line
This happens even on LIVE CD environments for Manjaro, Solus, and Ubuntu. Everything works as expected if I use PCIe as primary graphics