Linux Kernel 6 seems to be incompatible with the vfio_pci module needed for PCI passthrough

What’s your platform and the gpu model, and Linux version?

Arch, kernel 6.1.29, AMD RX550.

I am on Ubuntu 23.04. If I boot it with Linux 6.2.0-20, vfio-pci would not be loaded on boot. If I try to manually load it, it trashes my primary framebuffer, while gpu passthrough works as normal. I just can’t get any screen output from the host igpu.

If I boot it with Linux 5.19.0-40, everything works properly.

Thus, I believe it is either a bug in the Linux kernel, or in this Ubuntu distribution that has compatibility issue.

Unfortunately I have currently no means to test Ubuntu on my end, sorry.

Sounds a bit as if when vfio-pci is loaded it tries to bind something that is in the same IOMMU group as your iGPU. Have you set vfio-pci to bind anything at all at this stage, in the kernel parameters or somewhere else, or are you just trying to load the module without binding anything? Apart from that I can not really help further, passthrough with iGPUs seems like trouble, since I have heart from multiple people now that they run into problems with it.

I use boot parameter to set vfio ids.
I use igpu (7950x) for the host, Rtx 4070 ti for the vm. I can verify the vfio-pci does not grab the igpu when loaded, but it did unbind the igpu framebuffer somehow.

Then remove the boot parameter and boot without any passthrough devices and try to load the module. This way you can see if there is a problem with the module or with your passthrough setup.

OK, I removes the boot parameter, modprobe vfio-pci alone seems to be fine. However, it still freezes the host display when I start the VM which will pass through the dgpu.

01:00.0 VGA compatible controller: NVIDIA Corporation AD104 [GeForce RTX 4070 Ti] (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device 1696
        Kernel driver in use: vfio-pci
        Kernel modules: nvidiafb, nouveau
01:00.1 Audio device: NVIDIA Corporation Device 22bc (rev a1)
        Subsystem: ZOTAC International (MCO) Ltd. Device 1696
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

........

17:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Raphael (rev c1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Raphael
        Kernel modules: amdgpu
17:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
        Subsystem: Micro-Star International Co., Ltd. [MSI] Rembrandt Radeon High Definition Audio Controller
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

dmesg
Note that “Console: switching to colour dummy device 80x25”, which looks abnormal.

[   81.864066] VFIO - User Level meta-driver version: 0.3
[   81.886087] [UFW BLOCK] IN=br0 OUT= MAC=33:33:00:00:00:01:18:e8:29:bb:1e:96:86:dd SRC=fe80:0000:0000:0000:1ae8:29ff:febb:1e96 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=439 TC=0 HOPLIMIT=1 FLOWLBL=978196 PROTO=UDP SPT=57573 DPT=10001 LEN=399
[   81.886121] [UFW BLOCK] IN=eth11 OUT= MAC=33:33:00:00:00:01:18:e8:29:bb:1e:96:86:dd SRC=fe80:0000:0000:0000:1ae8:29ff:febb:1e96 DST=ff02:0000:0000:0000:0000:0000:0000:0001 LEN=439 TC=0 HOPLIMIT=1 FLOWLBL=978196 PROTO=UDP SPT=57573 DPT=10001 LEN=399
[   89.834334] audit: type=1400 audit(1685379925.605:95): apparmor="STATUS" operation="profile_load" profile="unconfined" name="libvirt-e3b05049-8201-4447-a4b3-b310220cc5bd" pid=11556 comm="apparmor_parser"
[   89.918579] audit: type=1400 audit(1685379925.689:96): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-e3b05049-8201-4447-a4b3-b310220cc5bd" pid=11559 comm="apparmor_parser"
[   90.004717] audit: type=1400 audit(1685379925.777:97): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-e3b05049-8201-4447-a4b3-b310220cc5bd" pid=11563 comm="apparmor_parser"
[   90.056972] Console: switching to colour dummy device 80x25
[   90.057014] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[   90.182079] xhci_hcd 0000:14:00.0: remove, state 4
[   90.182084] usb usb10: USB disconnect, device number 1
[   90.182351] xhci_hcd 0000:14:00.0: USB bus 10 deregistered
[   90.182365] xhci_hcd 0000:14:00.0: remove, state 1
[   90.182367] usb usb2: USB disconnect, device number 1
[   90.182368] usb 2-4: USB disconnect, device number 2
[   90.236678] usb 2-6: USB disconnect, device number 3
[   90.515885] usb 2-7: USB disconnect, device number 4
[   90.620220] usb 2-8: USB disconnect, device number 5
[   90.623790] usb 2-9: USB disconnect, device number 6
[   90.683982] xhci_hcd 0000:14:00.0: USB bus 2 deregistered
[   90.933609] audit: type=1400 audit(1685379926.705:98): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="libvirt-e3b05049-8201-4447-a4b3-b310220cc5bd" pid=11607 comm="apparmor_parser"
[   91.028129] audit: type=1400 audit(1685379926.797:99): apparmor="STATUS" operation="profile_replace" info="same as current profile, skipping" profile="unconfined" name="libvirt-e3b05049-8201-4447-a4b3-b310220cc5bd" pid=11612 comm="apparmor_parser"
[   92.633732] vfio-pci 0000:03:11.7: enabling device (0000 -> 0002)
[   92.872297] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
[   92.872313] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[   92.872318] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x26@0xc1c
[   92.872319] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x27@0xd00
[   92.872320] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x25@0xe00
[   92.886262] vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x25@0x160
[   96.327612] vfio-pci 0000:14:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x002d address=0xe8900 flags=0x0000]
[  107.143833] ixgbe 0000:03:00.1 eth11: VF Reset msg received from vf 7
[  107.237152] ixgbe 0000:03:00.1 eth11: VF Reset msg received from vf 7

What you can see now is that the module and kernel work as expected and your problem lies elsewhere.

Please run the following script as root …

#!/bin/bash
shopt -s nullglob
for g in $(find /sys/kernel/iommu_groups/* -maxdepth 0 -type d | sort -V); do
    echo "IOMMU Group ${g##*/}:"
    for d in $g/devices/*; do
        echo -e "\t$(lspci -nns ${d##*/})"
    done;
done;

Here you go.

IOMMU Group 0:
        00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 1:
        00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 2:
        00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 3:
        00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 4:
        00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 5:
        00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 6:
        00:02.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14db]
IOMMU Group 7:
        00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 8:
        00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 9:
        00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14da]
IOMMU Group 10:
        00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 11:
        00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14dd]
IOMMU Group 12:
        00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
        00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
        00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e0]
        00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e1]
        00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e2]
        00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e3]
        00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e4]
        00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e5]
        00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e6]
        00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14e7]
IOMMU Group 14:
        01:00.0 VGA compatible controller [0300]: NVIDIA Corporation AD104 [GeForce RTX 4070 Ti] [10de:2782] (rev a1)
        01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:22bc] (rev a1)
IOMMU Group 15:
        02:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1aa] (rev 03)
IOMMU Group 16:
        03:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU Group 17:
        03:00.1 Ethernet controller [0200]: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 [8086:1528] (rev 01)
IOMMU Group 18:
        04:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
IOMMU Group 19:
        05:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 20:
        05:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 21:
        05:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 22:
        05:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 23:
        05:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
IOMMU Group 24:
        05:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f4] (rev 01)
        0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0c:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        0e:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
        12:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
        13:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU Group 25:
        05:0c.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        14:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f7] (rev 01)
IOMMU Group 26:
        05:0d.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f5] (rev 01)
        15:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43f6] (rev 01)
IOMMU Group 27:
        16:00.0 Non-Volatile memory controller [0108]: Intel Corporation Device [8086:f1aa] (rev 03)
IOMMU Group 28:
        17:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Raphael [1002:164e] (rev c1)
IOMMU Group 29:
        17:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller [1002:1640]
IOMMU Group 30:
        17:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
IOMMU Group 31:
        17:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b6]
IOMMU Group 32:
        17:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b7]
IOMMU Group 33:
        17:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h/19h HD Audio Controller [1022:15e3]
IOMMU Group 34:
        18:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15b8]
IOMMU Group 35:
        03:10.1 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)
IOMMU Group 36:
        03:10.3 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)
IOMMU Group 37:
        03:10.5 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)
IOMMU Group 38:
        03:10.7 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)
IOMMU Group 39:
        03:11.1 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)
IOMMU Group 40:
        03:11.3 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)
IOMMU Group 41:
        03:11.5 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)
IOMMU Group 42:
        03:11.7 Ethernet controller [0200]: Intel Corporation X540 Ethernet Controller Virtual Function [8086:1515] (rev 01)

This looks fine the IOMMU groups are correct. Can you do me a favor and shutdown the computer, unplug any monitors or dummy plugs from the NVIDIA GPU, then start the computer and only when it has completely booted up plug the monitors back into the NVIDIA card. Then try to start the passthrough VM.

Yea, tried that. Same behavior. As soon as vfio-pci grabs the NVIDIA card, the igpu also loose its framebuffer, which means I don’t get any display output for the host afterwards. I only get this issue with kernel 6.2.0. It is absolutely fine with 5.19.0.

My host Ubuntu is running without desktop interface. I probably don’t need boot parameters to grab the card at the boot time. Virsh will load vfio-pci and grab the card just fine.

1 Like

Sorry, I am out of ideas on this.

I used every Linux version Manajro used up till 6.1 and after that i am on Nobara and definetly used 6.2 until now i am on 6.3.4 / 5

it did not have any problems regarding vfio-pci binding.
i am on AM5 sadly still with a 4080.

maybe that information helps someone

Do you run desktop interface on the host?

you mean like gnome?
Sure my VM is looking-glass only :smiley:

Yes.

My host is the bare minimum Ubuntu Server without the graphic interface. Nothing really holds on my host igpu except the console display. I think that is why its framebuffer is destroyed once I load the vfio-pci driver.

I will wait for couple months to see if it is resolved automatically. I don’t have this issue on the 5.x kernel.

1 Like

I use Ubuntu 22.04, and since kernel 6.2 my second GPU is not bound to vfio-pci anymore, using the GRUB cmdline method.

The reason for this behavior is a change in Ubuntu kernel configuration. vfio-pci is built as a module from 6.2 on, like ArchLinux:

$ find /lib/modules -name vfio-pci.ko
/lib/modules/6.2.0-26-generic/kernel/drivers/vfio/pci/vfio-pci.ko
(older kernels are not listed)

Since 6.2 it is possible to use two digital audio outputs at the same time with one graphics card (HDMI/DP), so I had to resolve it.

The solution is to use the initramfs method to load vfio-pci early:

# echo "vfio-pci" >>/etc/initramfs-tools/modules
# update-initramfs -u -k 6.2.0-26-generic

or simply reference the running kernel with
# update-initramfs -u -k $(uname -r)

If you prefer a position-based approach instead of a vendor/device ID approach, this method supports it as well. You can do the following:

Add a script vfio.sh to /etc/initramfs-tools/scripts/init-top/:

#!/bin/sh

PREREQ=""

prereqs()
{
   echo "$PREREQ"
}

case $1 in
prereqs)
   prereqs
   exit 0
   ;;
esac

for dev in 0000:0a:00.0 0000:0a:00.1 
do 
 echo "vfio-pci" > /sys/bus/pci/devices/$dev/driver_override 
 echo "$dev" > /sys/bus/pci/drivers/vfio-pci/bind 
done

exit 0

This script assumes a GPU with two appearances in 0a:00 PCI space. You probably have to adapt it to your system. Any change to onboard configuration in BIOS will change the position, so this method usually makes sense for a use case with two graphics cards having the same vendor/product ID or frequent change of the graphics card using different models.

4 Likes