[SOLVED] PCIe passthrough Problem with two identical graphic card - ACS Kernel

Software & Operating Systems Linux
1

Hello guys, I have a problem.

I have two RTX 2080 and I have follow this tutorial I have recompile kernel with acs patch, I have add this value in my grub:
GRUB_CMDLINE_LINUX_DEFAULT=“quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction”

So now my IOMMU group are all different, like that:

IOMMU Group 0:
	00:00.0 Host bridge [0600]: Intel Corporation Device [8086:3e30] (rev 0a)
IOMMU Group 1:
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0a)
IOMMU Group 10:
	00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port 21 [8086:a32c] (rev f0)
IOMMU Group 11:
	00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a338] (rev f0)
IOMMU Group 12:
	00:1c.2 PCI bridge [0604]: Intel Corporation Device [8086:a33a] (rev f0)
IOMMU Group 13:
	00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a33c] (rev f0)
IOMMU Group 14:
	00:1c.5 PCI bridge [0604]: Intel Corporation Device [8086:a33d] (rev f0)
IOMMU Group 15:
	00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:a33f] (rev f0)
IOMMU Group 16:
	00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a305] (rev 10)
	00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
	00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
	00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU Group 17:
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GV104 [GeForce GTX 1180] [10de:1e87] (rev a1)
IOMMU Group 18:
	01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f8] (rev a1)
IOMMU Group 19:
	01:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1)
IOMMU Group 2:
	00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 0a)
IOMMU Group 20:
	01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
IOMMU Group 21:
	02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GV104 [GeForce GTX 1180] [10de:1e87] (rev a1)
IOMMU Group 22:
	02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f8] (rev a1)
IOMMU Group 23:
	02:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1)
IOMMU Group 24:
	02:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
IOMMU Group 25:
	04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 [144d:a808]
IOMMU Group 26:
	05:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:2142]
IOMMU Group 27:
	06:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU Group 28:
	07:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E2500 Gigabit Ethernet Controller [1969:e0b1] (rev 10)
IOMMU Group 29:
	08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU Group 3:
	00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU Group 30:
	09:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU Group 4:
	00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 5:
	00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
	00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 6:
	00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9560 [Jefferson Peak] [8086:a370] (rev 10)
IOMMU Group 7:
	00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 8:
	00:17.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 10)
IOMMU Group 9:
	00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a340] (rev f0)

Ok good, now the guys who create the ACS tutorial recommend:
to prevent issues in or booting the VM, I would assign all 4 functions/device IDs of the RTX cards to the VFIO-PCI driver.
and he put a screen about his /etc/initramfs-tools/modules:
:
so I do the same with my PCI-ID (note: my pci-id of my two rtx are the same at this moment, but are separated with different IOMMO group):
nano /etc/initramfs-tools/modules:

softdep nvidia pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:1e87
options vfio_pci ids=10de:10f8
options vfio_pci ids=10de:1ad8
options vfio_pci ids=10de:1ad9
options vfio_pci ids=10de:10f8,10de:1ad8,10de:1ad9
vfio_pci ids=10de:10f8,10de:1ad8,10de:1ad9
vfio_pci
nvidia

Ok so now i continue the normal tutorial:
nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9

nano /etc/modprobe.d/nvidiagpu.conf

softdep nvidia pre: vfio vfio_pci

nano /etc/modprobe.d/vfio_pci.conf

options vfio_pci ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9

ok now I run:
grub-mkconfig; sudo update-grub; sudo update-initramfs -k all -c

I reboot on my grub and… Black screen…
I have loss my principal the GPU
I think it’s works, it will be shutdown the fist GPU but i just want shutdown the second not the one.

So how I can do that ? My PCI-ID are the same.
Any idea? I’m on Intel/Xubuntu 18.04

2

This guide shows how to reassign specific cards, so your u don’t do it by device id

3

Thanks wendell for your help, i have follow your guide, now my graphic card seem to be OK to passthrough , but i have another problem.

I create my virtual machine on QMENU with your recommendations (Q35 chipset and OVMF), i can see my RTX in the peripheral devices but when i want install the driver i have an error

ScreenShot%20On%20Linux%20at%2019-07-08
ScreenShot On Linux at 19-07-08.png793×595 99.3 KB

I have make all windows update.
Any ideas?
thanks

4

You need the dch driver if this is windows 1903 … from the Nvidia website

5

Ok I have install the good driver but the problem is always here:

ScreenShot%20On%20Linux%20at%2021-15-04
ScreenShot On Linux at 21-15-04.png750×530 43.1 KB

I have one triangle on the RTX and two on PCI controller.
As you can see I have installed the driver :thinking:
and what driver i need to install the BUS PCI ?
Windows lock me in 800X600 resolution.
thanks for your help.

6

Ok problem solved with this tread: Configuring a headless Linux OS installation strictly for virtualizing then managing a Windows installation?

thanks again.

2 Likes