VFIO PCIe-NIC Quad Ports Passthrough

Operating Systems & Open Source Linux
1

HI,

I bought a 10Gtek CNA/NIC with Intel 82580 Chipset.
The problem is that only one (of four) NICs uses the vfio-driver.
My goal is that all four NICs are accessible from the Guests only.
I hope you can help me fix this problem.

I added the following to grub.cfg …

linux /@/kernels/4.14.33-FC.01/vmlinuz root=rpool/ROOT/gentoo by=id elevator=noop quiet logo.nologo triggers=zfs iommu=pt iommu=1

Output of dmesg …

[    0.638212] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    0.638212] AMD-Vi: Interrupt remapping enabled
[    0.638277] AMD-Vi: Lazy IO/TLB flushing enabled
[    5.080354] VFIO - User Level meta-driver version: 0.3
[    5.203225] vfio_pci: add [8086:150e[ffff:ffff]] class 0x000000/00000000

/etc/modprobe.d/vfio.conf …

options vfio-pci ids=8086:150e

There was no output when running dmesg | grep ecap so I assumed to add the following file.

/etc/modprobe.d/iommu_unsafe_interrupts.conf …

options vfio_iommu_type1 allow_unsafe_interrupts=1

Output of lsmod …

vfio_pci               45056  0
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       24576  0
vfio                   28672  2 vfio_iommu_type1,vfio_pci
irqbypass              16384  2 kvm,vfio_pci

List of all IOMMUs …

IOMMU Group 0 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge [1002:5a14] (rev 02)
IOMMU Group 1 00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0) [1002:5a16]
IOMMU Group 10 00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383] (rev 40)
IOMMU Group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40)
IOMMU Group 12 00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40)
IOMMU Group 13 00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
IOMMU Group 14 00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU Group 14 00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU Group 15 01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
IOMMU Group 15 01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
IOMMU Group 16 02:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 17 02:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 18 02:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 19 02:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 2 00:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0 PCI to PCI bridge (PCI Express GFX port 1) [1002:5a17]
IOMMU Group 20 03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller [1b21:1343]
IOMMU Group 21 04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller [1b21:1142]
IOMMU Group 22 05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 3 00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 0) [1002:5a18]
IOMMU Group 4 00:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 2) [1002:5a1a]
IOMMU Group 5 00:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 3) [1002:5a1b]
IOMMU Group 6 00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40)
IOMMU Group 7 00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU Group 7 00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU Group 8 00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
IOMMU Group 8 00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
IOMMU Group 9 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42)

Here you can see the drivers that are in use for the 10Gtek CNA/NIC …

    02:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
    	Subsystem: Intel Corporation Ethernet Server Adapter I340-T4 [8086:12a1]
    	Kernel driver in use: igb
    	Kernel modules: igb
    02:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
    	Subsystem: Intel Corporation Ethernet Server Adapter I340-T4 [8086:12a1]
    	Kernel driver in use: igb
    	Kernel modules: igb
    02:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
    	Subsystem: Intel Corporation Ethernet Server Adapter I340-T4 [8086:12a1]
    	Kernel driver in use: igb
    	Kernel modules: igb
    02:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
    	Subsystem: Intel Corporation Ethernet Server Adapter I340-T4 [8086:12a1]
    	Kernel driver in use: vfio-pci
    	Kernel modules: igb

I should also mention that the NIC on my Mobo uses the same driver …

05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
	Subsystem: ASUSTeK Computer Inc. I211 Gigabit Network Connection [1043:85f0]
	Kernel driver in use: igb
	Kernel modules: igb

I am using Gentoo with Bliss-Kernel …

Linux mypc 4.14.33-FC.01 #2 SMP Mon Apr 9 10:36:59 EDT 2018 x86_64 AMD FX(tm)-8350 Eight-Core Processor AuthenticAMD GNU/Linux

I tried to put all 0000:02:00.* in driver unbind and vfio-pci bind but every time I reboot its the same problem again.

# echo "8086 150e" > /sys/bus/pci/drivers/vfio-pci/new_id 
# echo "0000:02:00.0" > /sys/bus/pci/devices/0000\:02\:00.0/driver/unbind 
# echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
# echo "0000:02:00.1" > /sys/bus/pci/devices/0000\:02\:00.1/driver/unbind 
# echo "0000:02:00.1" > /sys/bus/pci/drivers/vfio-pci/bind
# echo "0000:02:00.2" > /sys/bus/pci/devices/0000\:02\:00.2/driver/unbind 
# echo "0000:02:00.2" > /sys/bus/pci/drivers/vfio-pci/bind

What I am doing wrong and can u halp me ?

2

This is only a runtime config change.

Just create a unit file that is a oneshot to run a script that does this every time you boot.

I don’t know why the other three aren’t binding on boot. You could do a rd.driver.pre hook to force it to try vfio-pci before loading the igb driver.

3

While tried to create a initramfs the following error cam up …

dracut: dracut module ‘vfio_pci’ cannot be found or installed.

I am not sure why this happen, maybe it has some to do with the bliss-kernel (third party) I am using.

Is there an other way to create a rd.driver.pre hook ?

4

Hmm maybe. Check if bliss-kernel even includes the modules.

5

$ zcat /proc/config.gz | grep -i vfio
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO_VIRQFD=m
CONFIG_VFIO=m

CONFIG_VFIO_NOIOMMU is not set

CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_VFIO_PCI_MMAP=y
CONFIG_VFIO_PCI_INTX=y
CONFIG_VFIO_PCI_IGD=y
CONFIG_VFIO_MDEV=m
CONFIG_VFIO_MDEV_DEVICE=m
CONFIG_KVM_VFIO=y

Still not sure why bliss-kernel wont let me use dracut.

Now I will create my own kernel and hope for the best.
But this will take some time cause I run into compatibility problems cause of no bleeding-edge zfs on Gentoo, wierd. I need to compile a new zfs version via git so I can compile an up to date kernel-source (>4.19).