[SOLVED] PCIe passthrough Problem with two identical graphic card - ACS Kernel

**jeremie_duval** · May 30, 2019, 8:55pm

Hello guys, I have a problem.

I have two RTX 2080 and I have follow this tutorial I have recompile kernel with acs patch, I have add this value in my grub:
GRUB_CMDLINE_LINUX_DEFAULT=“quiet intel_iommu=on iommu=pt pcie_acs_override=downstream,multifunction”

So now my IOMMU group are all different, like that:

IOMMU Group 0:
	00:00.0 Host bridge [0600]: Intel Corporation Device [8086:3e30] (rev 0a)
IOMMU Group 1:
	00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 0a)
IOMMU Group 10:
	00:1b.4 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port 21 [8086:a32c] (rev f0)
IOMMU Group 11:
	00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a338] (rev f0)
IOMMU Group 12:
	00:1c.2 PCI bridge [0604]: Intel Corporation Device [8086:a33a] (rev f0)
IOMMU Group 13:
	00:1c.4 PCI bridge [0604]: Intel Corporation Device [8086:a33c] (rev f0)
IOMMU Group 14:
	00:1c.5 PCI bridge [0604]: Intel Corporation Device [8086:a33d] (rev f0)
IOMMU Group 15:
	00:1c.7 PCI bridge [0604]: Intel Corporation Device [8086:a33f] (rev f0)
IOMMU Group 16:
	00:1f.0 ISA bridge [0601]: Intel Corporation Device [8086:a305] (rev 10)
	00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
	00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
	00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
IOMMU Group 17:
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GV104 [GeForce GTX 1180] [10de:1e87] (rev a1)
IOMMU Group 18:
	01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f8] (rev a1)
IOMMU Group 19:
	01:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1)
IOMMU Group 2:
	00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) [8086:1905] (rev 0a)
IOMMU Group 20:
	01:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
IOMMU Group 21:
	02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GV104 [GeForce GTX 1180] [10de:1e87] (rev a1)
IOMMU Group 22:
	02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f8] (rev a1)
IOMMU Group 23:
	02:00.2 USB controller [0c03]: NVIDIA Corporation Device [10de:1ad8] (rev a1)
IOMMU Group 24:
	02:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device [10de:1ad9] (rev a1)
IOMMU Group 25:
	04:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 [144d:a808]
IOMMU Group 26:
	05:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:2142]
IOMMU Group 27:
	06:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU Group 28:
	07:00.0 Ethernet controller [0200]: Qualcomm Atheros Killer E2500 Gigabit Ethernet Controller [1969:e0b1] (rev 10)
IOMMU Group 29:
	08:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU Group 3:
	00:08.0 System peripheral [0880]: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th Gen Core Processor Gaussian Mixture Model [8086:1911]
IOMMU Group 30:
	09:00.0 Ethernet controller [0200]: Intel Corporation 82574L Gigabit Network Connection [8086:10d3]
IOMMU Group 4:
	00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
IOMMU Group 5:
	00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10)
	00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
IOMMU Group 6:
	00:14.3 Network controller [0280]: Intel Corporation Wireless-AC 9560 [Jefferson Peak] [8086:a370] (rev 10)
IOMMU Group 7:
	00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
IOMMU Group 8:
	00:17.0 RAID bus controller [0104]: Intel Corporation SATA Controller [RAID mode] [8086:2822] (rev 10)
IOMMU Group 9:
	00:1b.0 PCI bridge [0604]: Intel Corporation Device [8086:a340] (rev f0)

Ok good, now the guys who create the ACS tutorial recommend:
to prevent issues in or booting the VM, I would assign all 4 functions/device IDs of the RTX cards to the VFIO-PCI driver.
and he put a screen about his /etc/initramfs-tools/modules:
:
so I do the same with my PCI-ID (note: my pci-id of my two rtx are the same at this moment, but are separated with different IOMMO group):
nano /etc/initramfs-tools/modules:

softdep nvidia pre: vfio vfio_pci

vfio
vfio_iommu_type1
vfio_virqfd
options vfio_pci ids=10de:1e87
options vfio_pci ids=10de:10f8
options vfio_pci ids=10de:1ad8
options vfio_pci ids=10de:1ad9
options vfio_pci ids=10de:10f8,10de:1ad8,10de:1ad9
vfio_pci ids=10de:10f8,10de:1ad8,10de:1ad9
vfio_pci
nvidia

Ok so now i continue the normal tutorial:
nano /etc/modules

vfio
vfio_iommu_type1
vfio_pci ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9

nano /etc/modprobe.d/nvidiagpu.conf

softdep nvidia pre: vfio vfio_pci

nano /etc/modprobe.d/vfio_pci.conf

options vfio_pci ids=10de:1e87,10de:10f8,10de:1ad8,10de:1ad9

ok now I run:
grub-mkconfig; sudo update-grub; sudo update-initramfs -k all -c

I reboot on my grub and… Black screen…
I have loss my principal the GPU
I think it’s works, it will be shutdown the fist GPU but i just want shutdown the second not the one.

So how I can do that ? My PCI-ID are the same.
Any idea? I’m on Intel/Xubuntu 18.04

**wendell** · May 30, 2019, 10:54am

This guide shows how to reassign specific cards, so your u don’t do it by device id

VFIO in 2019 -- Pop!_OS How-To (General Guide though) [DRAFT] Wikis & How-to Guides

The Ultimate (Same GPU) VFIO Passthrough Guide for 2019 (now notes on getting samey graphics cards working) What are we doing? What is VFIO/Passthrough? If you are an absolute newbie – all the hub-ubb is about being able to run Linux as your primary OS, with a Windows virtual machine running under Linux for everything else. The key aspect here, though, is that you are running a computer with two graphics cards and full control over one graphics card is given to the Windows virtual machine. That means you can run games and GPU-accelerated apps (like the Adobe creative suite) just like they were running on a full computer, but in a virtual machine. Windows in a VM gives you great flexibility for backups, snapshots and versioning. It gives you better control over your privacy and what the operating system might be doing and it means that Windows only runs on your computer when you have to. I’ve been running VFIO for about 10 years and I have found that I rarely even need Windows for much of anything in my personal computation needs these days (even for gaming). Still, there is some usefulness in having the ‘security blanket’ of being able to get to your old Windows install exactly as it was before you switched operating systems without having to reboot. The System (76) For this guide, we are using Pop!_OS from System76 as our Linux distro of choice. I got to borrow Eric Raymond’s new computer, a monster 32-core Threadripper 2990WX system and, for this guide, it is equipped with dual Nvidia RTX 2080s from Gigabyte (Eric ultimately gets a WX7100 graphics card). Our Hardware This is the System76 Thelio Major r1 It has: AMD Threadripper 2990WX (2x) Gigabyte RTX 2080 GPUs (4x) 16gb Dual Rank Kingston HyperX 2933 DDR4 (16-18-18-36 1T @ 2933) Samsung SM97x 2tb NVMe Intel 750 1.2tb NVMe (guest OS) Arctic Cooling fans With Lutris/DXVK and Steam on Linux – I like the idea of having the same GPU horsepower for both guest and host. This system? It has two Gigabyte RTX…

**jeremie_duval** · May 30, 2019, 5:23pm

Thanks wendell for your help, i have follow your guide, now my graphic card seem to be OK to passthrough , but i have another problem.

I create my virtual machine on QMENU with your recommendations (Q35 chipset and OVMF), i can see my RTX in the peripheral devices but when i want install the driver i have an error

I have make all windows update.
Any ideas?
thanks

**wendell** · May 30, 2019, 6:49pm

You need the dch driver if this is windows 1903 … from the Nvidia website

**jeremie_duval** · May 30, 2019, 7:45pm

Ok I have install the good driver but the problem is always here:

I have one triangle on the RTX and two on PCI controller.
As you can see I have installed the driver
and what driver i need to install the BUS PCI ?
Windows lock me in 800X600 resolution.
thanks for your help.

**jeremie_duval** · May 30, 2019, 8:54pm

Ok problem solved with this tread: Configuring a headless Linux OS installation strictly for virtualizing then managing a Windows installation?

thanks again.