Passthrough does not work

Software & Operating Systems VFIO
1

Hello.

I’m trying to enable the passthrough on my graphic card (nvidia RTX 2080 ti on CPU I9 - 9900k) on the debian 10. These are the commands that I gave :

nano /etc/initramfs-tools/modules

adding this line :

vfio_pci ids= 10de:1e04,10de:10f7

below you can see how I have got that numbers :

ziomario@DESKTOP-N9UN2H3:/home/mariozio# lspci -nn | grep 01:00.

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)

and then :

update-initramfs -u
update-initramfs -u -k all

and

nano /etc/modprobe.d/blacklist.conf

adding this line :

blacklist nouveau
options nouveau modeset-0

the tutorial that I’ve followed does not explain if I should modify also the file /etc/default/grub. Anyway I tried,adding this line :

GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash intel_iommu=on iommu=pt”

instead of the old one,that was : GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash”

and then I did again :

update-initramfs -u
update-initramfs -u -k all

but the result does not change. During the boot I see the blinking cursor hanging on the screen and nothing happens anymore. I have even removed all the changes that I made and I did again :

update-initramfs -u
update-initramfs -u -k all

to understand what’s the problem and I seen that if I add this line :

vfio_pci ids= 10de:1e04,10de:10f7

and then I reboot,it freezes. Do u have some suggestion for me ? thanks.

2

First you need to add the lines to grub, then you need to update grub. Then you can move on to doing vfio and updating your initramfs.

See-
https://wiki.debian.org/VGAPassthrough
“On Intel platforms it is necessary to add intel_iommu=on on the kernel commandline (add in to GRUB_CMDLINE_LINUX_DEFAULT in /etc/default/grub and run sudo update-grub ).”

3

yes,thanks. I did it,already. but it does not work anyway.

4

whats the output of lspci -nnv

1 Like
5

There should be no space. in the modprobe.d/*.conf file.

It should look like this:

vfio_pci ids=10de:1e04,10de:10f7

Additionally, if you want to add the vfio_pci config to the kernel command line, you need to use the following:

vfio_pci.ids=10de:1e04,10de:10f7
6

this is the correct sintax :

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1e04,10de:10f7

if I do : vfio_pci ids=10de:1e04,10de:10f7

I get this error :

libkmod: ERROR …/libkmod/libkmod-config.c:656 kmod_config_parse: /etc/modprobe.d/vfio.conf line 1: ignoring bad line starting with ‘vfio-pci’

7

What is the output of lspci -nnv

Need to find out of what driver is being loaded for the gpu

8

ziomario@DESKTOP-N9UN2H3:/home/mariozio# lspci -nnv

00:00.0 Host bridge [0600]: Intel Corporation 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [8086:3e30] (rev 0d)
Subsystem: Gigabyte Technology Co., Ltd 8th Gen Core 8-core Desktop Processor Host Bridge/DRAM Registers [Coffee Lake S] [1458:5000]
Flags: bus master, fast devsel, latency 0
Capabilities: [e0] Vendor Specific Information: Len=10 <?>
Kernel driver in use: skl_uncore

00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 0d) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 16
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: 00003000-00003fff
Memory behind bridge: 62000000-650fffff
Prefetchable memory behind bridge: 0000000050000000-00000000600fffff
Capabilities: [88] Subsystem: Gigabyte Technology Co., Ltd Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) [1458:5000]
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Capabilities: [d94] #19
Kernel driver in use: pcieport

00:02.0 Display controller [0380]: Intel Corporation Device [8086:3e98] (rev 02)
Subsystem: Gigabyte Technology Co., Ltd Device [1458:d000]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at 66000000 (64-bit, non-prefetchable) [size=16M]
Memory at 40000000 (64-bit, prefetchable) [size=256M]
I/O ports at 4000 [size=64]
Capabilities: [40] Vendor Specific Information: Len=0c <?>
Capabilities: [70] Express Root Complex Integrated Endpoint, MSI 00
Capabilities: [ac] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [d0] Power Management version 2
Capabilities: [100] Process Address Space ID (PASID)
Capabilities: [200] Address Translation Service (ATS)
Capabilities: [300] Page Request Interface (PRI)

00:12.0 Signal processing controller [1180]: Intel Corporation Cannon Lake PCH Thermal Controller [8086:a379] (rev 10)
Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH Thermal Controller [1458:8888]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at 6713d000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Kernel driver in use: intel_pch_thermal
Kernel modules: intel_pch_thermal

00:14.0 USB controller [0c03]: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller [8086:a36d] (rev 10) (prog-if 30 [XHCI])
Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH USB 3.1 xHCI Host Controller [1458:5007]
Flags: bus master, medium devsel, latency 0, IRQ 125
Memory at 67120000 (64-bit, non-prefetchable) [size=64K]
Capabilities: [70] Power Management version 2
Capabilities: [80] MSI: Enable+ Count=1/8 Maskable- 64bit+
Capabilities: [90] Vendor Specific Information: Len=14 <?>
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

00:14.2 RAM memory [0500]: Intel Corporation Cannon Lake PCH Shared SRAM [8086:a36f] (rev 10)
Subsystem: Intel Corporation Cannon Lake PCH Shared SRAM [8086:7270]
Flags: bus master, fast devsel, latency 0
Memory at 67136000 (64-bit, non-prefetchable) [size=8K]
Memory at 6713c000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [80] Power Management version 3

00:16.0 Communication controller [0780]: Intel Corporation Cannon Lake PCH HECI Controller [8086:a360] (rev 10)
Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH HECI Controller [1458:1c3a]
Flags: bus master, fast devsel, latency 0, IRQ 127
Memory at 6713b000 (64-bit, non-prefetchable) [size=4K]
Capabilities: [50] Power Management version 3
Capabilities: [8c] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [a4] Vendor Specific Information: Len=14 <?>
Kernel driver in use: mei_me
Kernel modules: mei_me

00:17.0 SATA controller [0106]: Intel Corporation Cannon Lake PCH SATA AHCI Controller [8086:a352] (rev 10) (prog-if 01 [AHCI 1.0])
Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH SATA AHCI Controller [1458:b005]
Flags: bus master, 66MHz, medium devsel, latency 0, IRQ 124
Memory at 67134000 (32-bit, non-prefetchable) [size=8K]
Memory at 6713a000 (32-bit, non-prefetchable) [size=256]
I/O ports at 4090 [size=8]
I/O ports at 4080 [size=4]
I/O ports at 4060 [size=32]
Memory at 67139000 (32-bit, non-prefetchable) [size=2K]
Capabilities: [80] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [70] Power Management version 3
Capabilities: [a8] SATA HBA v1.0
Kernel driver in use: ahci
Kernel modules: ahci

00:1b.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a340] (rev f0) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
Capabilities: [40] Express Root Port (Slot-), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH PCI Express Root Port [1458:5001]
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport

00:1c.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a338] (rev f0) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=03, subordinate=03, sec-latency=0
Capabilities: [40] Express Root Port (Slot-), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH PCI Express Root Port [1458:5001]
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport

00:1d.0 PCI bridge [0604]: Intel Corporation Cannon Lake PCH PCI Express Root Port [8086:a330] (rev f0) (prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0
Bus: primary=00, secondary=04, subordinate=04, sec-latency=0
I/O behind bridge: 00005000-00005fff
Memory behind bridge: 3f800000-3f9fffff
Prefetchable memory behind bridge: 000000003fa00000-000000003fbfffff
Capabilities: [40] Express Root Port (Slot+), MSI 00
Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
Capabilities: [90] Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH PCI Express Root Port [1458:5001]
Capabilities: [a0] Power Management version 3
Kernel driver in use: pcieport

00:1f.0 ISA bridge [0601]: Intel Corporation Z390 Chipset LPC/eSPI Controller [8086:a305] (rev 10)
Subsystem: Gigabyte Technology Co., Ltd Z390 Chipset LPC/eSPI Controller [1458:5001]
Flags: bus master, medium devsel, latency 0

00:1f.3 Audio device [0403]: Intel Corporation Cannon Lake PCH cAVS [8086:a348] (rev 10)
Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH cAVS [1458:a0c3]
Flags: bus master, fast devsel, latency 32, IRQ 129
Memory at 67130000 (64-bit, non-prefetchable) [size=16K]
Memory at 67000000 (64-bit, non-prefetchable) [size=1M]
Capabilities: [50] Power Management version 3
Capabilities: [80] Vendor Specific Information: Len=14 <?>
Capabilities: [60] MSI: Enable+ Count=1/1 Maskable- 64bit+
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

00:1f.4 SMBus [0c05]: Intel Corporation Cannon Lake PCH SMBus Controller [8086:a323] (rev 10)
Subsystem: Gigabyte Technology Co., Ltd Cannon Lake PCH SMBus Controller [1458:5001]
Flags: medium devsel, IRQ 16
Memory at 67138000 (64-bit, non-prefetchable) [size=256]
I/O ports at efa0 [size=32]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801

00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller [8086:a324] (rev 10)
Subsystem: Intel Corporation Cannon Lake PCH SPI Controller [8086:7270]
Flags: fast devsel
Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

00:1f.6 Ethernet controller [0200]: Intel Corporation Ethernet Connection (7) I219-V [8086:15bc] (rev 10)
Subsystem: Gigabyte Technology Co., Ltd Ethernet Connection (7) I219-V [1458:e000]
Flags: bus master, fast devsel, latency 0, IRQ 123
Memory at 67100000 (32-bit, non-prefetchable) [size=128K]
Capabilities: [c8] Power Management version 3
Capabilities: [d0] MSI: Enable+ Count=1/1 Maskable- 64bit+
Kernel driver in use: e1000e
Kernel modules: e1000e

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU102 [GeForce RTX 2080 Ti] [10de:1e04] (rev a1) (prog-if 00 [VGA controller])
Subsystem: ZOTAC International (MCO) Ltd. TU102 [GeForce RTX 2080 Ti] [19da:2503]
Flags: bus master, fast devsel, latency 0, IRQ 128
Memory at 64000000 (32-bit, non-prefetchable) [size=16M]
Memory at 50000000 (64-bit, prefetchable) [size=256M]
Memory at 62000000 (64-bit, prefetchable) [size=32M]
I/O ports at 3000 [size=128]
[virtual] Expansion ROM at 000c0000 [disabled] [size=128K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Legacy Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] #19
Capabilities: [bb0] #15
Kernel driver in use: nvidia
Kernel modules: nvidia

01:00.1 Audio device [0403]: NVIDIA Corporation TU102 High Definition Audio Controller [10de:10f7] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 High Definition Audio Controller [19da:2503]
Flags: bus master, fast devsel, latency 0, IRQ 17
Memory at 65080000 (32-bit, non-prefetchable) [size=16K]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel

01:00.2 USB controller [0c03]: NVIDIA Corporation TU102 USB 3.1 Controller [10de:1ad6] (rev a1) (prog-if 30 [XHCI])
Subsystem: ZOTAC International (MCO) Ltd. TU102 USB 3.1 Controller [19da:2503]
Flags: bus master, fast devsel, latency 0, IRQ 126
Memory at 60000000 (64-bit, prefetchable) [size=256K]
Memory at 60040000 (64-bit, prefetchable) [size=64K]
Capabilities: [68] MSI: Enable+ Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Power Management version 3
Capabilities: [100] Advanced Error Reporting
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci

01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU102 UCSI Controller [10de:1ad7] (rev a1)
Subsystem: ZOTAC International (MCO) Ltd. TU102 UCSI Controller [19da:2503]
Flags: bus master, fast devsel, latency 0, IRQ 11
Memory at 65084000 (32-bit, non-prefetchable) [size=4K]
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [b4] Power Management version 3
Capabilities: [100] Advanced Error Reporting

9

Also,someone told me that I need to run a kernel version better than 4.19. Actually I have :

4.19.0-6-amd64 #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) x86_64 GNU/Linux

is that true ?

10

New kernel will help but that is not what your issue is.

Your issue is the vfio config, its not set correctly, look at your gpu

Kernel driver in use: nvidia

You have not correctly isolated the gpu, and with the RTX cards you need to passthrough everything on it, Video audio and usb controller for the gpu

and did you rebuild your init correctly and reboot after

1 Like
11

run sudo update-initramfs -u and reboot

12

Yeah, you need options in front. sorry, I wrote that from mobile :confused:

Not true.

Based on this, you need a softdep.

For completeness, put this in /etc/modprobe.d/vfio.conf:

softdep vfio-pci pre: nvidia
softdep vfio-pci pre: nouveau

This forces vfio to load before the nvidia driver, and it’ll grab it.

For reference, mine looks like this:

$ cat /etc/modprobe.d/vfio.conf
options vfio-pci ids=10de:1b06,10de:10ef,1b73:1100
softdep vfio-pci pre: nouveau
softdep vfio-pci pre: nvidia
softdep vfio-pci pre: amdgpu
1 Like
13

After you edit the files, dont forget to run sudo update-initramfs -u

14

i don’t think there’s supposed to be a space between the equal signs

15

let’s recap :

/etc/initramfs-tools/modules :

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

/etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1e04,10de:10f7,10de:1ad6
softdep vfio-pci pre: nvidia
softdep vfio-pci pre: nouveau

/etc/default/grub :

GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash intel_iommu=on”

/etc/modprobe.d/blacklist.conf

blacklist nouveau
options nouveau modeset-0
blacklist radeon
blacklist amdg

is that all good ?

16

That looks good, but might need more.
The idea the guys are trying to convey, is that the GPU needs to be isolated from the host, so we need to change:

Into something like:
Kernel driver in use: vfio-pci
Kernel modules: vfio-pci

Or stub or something, for the three parts of the card (graphics, sound, and usb for etc)

We might have to use an additional file instead of the mod probe vfio file

1 Like
17

ok. what should I do then ? I’ve modified the configuration exactly as I wrote in the recap post and debian has been able to boot correctly.but when I gave these commands :

lsmod | grep vfio
dmesg | grep VFIO

I seen no output. so it does not work.

18

we want to get records in:
lspci -knn | grep vfio

(or scroll down it without grep) to check the three id’s have been snagged by the vfio driver (or blocked from nvidia/nouvau)

19

I prevously couldn’t get Fedora to grab the card I needed, so found a guide:

found redit post from https://www.reddit.com/r/VFIO/comments/a8swkd/fedora_29_mkinitcpio_doesnt_exist_or/

#echo “vfio” > /etc/modules-load.d/vfio.conf
#echo “vfio-pci” > /etc/modules-load.d/vfio-pci.conf
#echo “vfio_iommu_type1” > /etc/modules-load.d/vfio_iommu_type1.conf

Worth a shot?

20

lspci -knn | grep vfio gives no output at all.