I used the ACS patch located here: https://queuecumber.gitlab.io/linux-acs-override/ for kernel 4.19 and put the command pcie_acs_override=downstream in the default grub yet my GPU is still grouped with several other devices.
IOMMU Group 12 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d5] (rev 01)
IOMMU Group 12 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c8] (rev 01)
IOMMU Group 12 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c6] (rev 01)
IOMMU Group 12 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 12 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 12 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
Otherwise, do you have another available slot to try the GPU in? You could switch it with the host GPU, and if possible switch it in the bios/uefi so you actually still boot off of the correct GPU.
If moving slots does not work, then you are stuck till you get a different motherboard.
Unfortunately my bios has no option to boot off the GPU in my second PCI slot, it just defaults to the one closest to the CPU. I actually tried a setup like that a couple of weeks ago and was basically told that no bios has the option to switch between GPUs in the PCI slot as the primary display and that they only have to option to switch between devices in the PCI slots and the on board GPU (which my mobo doesn’t have). I don’t know how true this is but it’s what I was told. I can completely switch my GPUs around the two PCI slots I have but I’m geussing I’m just going to run into the same problem trying to passthrough the GPU in the second slot again?
I do have an PCI X4 slot in the middle of my two X16 slots but I’d have to get an adapter to fix my GPU in there.
Is sometimes possible to pass through the boot GPU but it adds another layer of config and complexity.
Edit- For setting a manual vBIOS rom, unless you can find one that matches your card exactly, dump it yourself. It is not very hard to do, well as compared to the rest of settting up passthrough.
Wow, thank you. I wish I would have known this before. I got a whole new case to fit two full GPUs in it because before I had to use a single slot cheap GT 710 in my second slot because that was the only thing that could fit and I wanted to passthrough my R9 390 in my main PCI slot. Using this mATX mobo for GPU passthrough so far has been a pain in the ass.
I must ask, what is your MB and CPU?
From what little is posted i would guess AMD Zen based setup and you have your GPU in a chipset attached slot.
In that case i would set pcie_acs_override=id:1022:43c6.
This should be enough to separate chipsets connected devices and pcie slots to different iommu groups.
Using downstream or multifunction is the last resort.
Still I would put the GPU into the CPU connected pcie slot.
And if you have only one try to disable CSMT in your bios, i saw several boards that changed its behaviour to boot from chipset connected GPU.
You will need EFI enabled GPU for this.
My Mobo is a Tuff B450M-PLUS and my CPU is a Ryzen 1800x. The two GPU I have in my system are a GTX 760 for the host OS and a R9 390 for the guest. I will try the grub setting you posted. I managed to get the GPU passed through using ‘pcie_acs_override=multifunction’ but QEMU using virt-manager refuses to boot into my Windows 10 iso. I’m pretty sure it’s because I have CSM enabled in the bios but unfortunately if I turn that setting off my bios defaults to the R9 390 and Ubuntu tries to boot off of it. Pretty sure my R9 390 is EFI capable and that’s the reason my BIOS defaults to it with CSM off, or at least that’s what I’m guessing. I guess I could just use the R9 390 as the host and the GTX 760 as the guest but that really wouldn’t be preferable because I want the stronger card on the Windows VM.
As I suspected. Your R9 390 is connected to chipset which is only PCIe 2.0 x4.
You could change the cards and disable CSM. For whatever reason, for Zen, if you disable CSM the boot GPU is the chipset one. My guess is because it has lover PCIe address. If your GTX 760 has EFI bios system will be booted from this card.
Than you would not need ACS patch at all.
I would also try pcie_acs_override=id:1022:43c6 instead of multifunction. And pass both GPU and its HDMI audio card connected via pcie-root-port or ioh3420.
I switched around my GPUs disabled CSM and now my iommu groupings are perfect. My GPU is separated from every other device in my system even without the ACS patch. I am still having issues with booting into my Windows 10 iso though. QEMU will reset the VM everytime it tries to boot into the Windows installation. I thought this was because CSM was enabled but it looks like that isn’t the case. I think I am going to make another thread about this problem.