PCI-NIC at IOMMU Group 0 with new C7HWIFI Mobo

Suleiman · March 10, 2019, 1:25am

I am confused.
Each NIC on my pci-card had its own group when I was using my old computer, as you can see in one of my oder post.

VFIO PCIe-NIC Quad Ports Passthrough Linux

HI, I bought a 10Gtek CNA/NIC with Intel 82580 Chipset. The problem is that only one (of four) NICs uses the vfio-driver. My goal is that all four NICs are accessible from the Guests only. I hope you can help me fix this problem. I added the following to grub.cfg … linux /@/kernels/4.14.33-FC.01/vmlinuz root=rpool/ROOT/gentoo by=id elevator=noop quiet logo.nologo triggers=zfs iommu=pt iommu=1 Output of dmesg … [ 0.638212] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40 [ 0.638212] AMD-Vi: Interrupt remapping enabled [ 0.638277] AMD-Vi: Lazy IO/TLB flushing enabled [ 5.080354] VFIO - User Level meta-driver version: 0.3 [ 5.203225] vfio_pci: add [8086:150e[ffff:ffff]] class 0x000000/00000000 /etc/modprobe.d/vfio.conf … options vfio-pci ids=8086:150e There was no output when running dmesg | grep ecap so I assumed to add the following file. /etc/modprobe.d/iommu_unsafe_interrupts.conf … options vfio_iommu_type1 allow_unsafe_interrupts=1 Output of lsmod … vfio_pci 45056 0 vfio_virqfd 16384 1 vfio_pci vfio_iommu_type1 24576 0 vfio 28672 2 vfio_iommu_type1,vfio_pci irqbypass 16384 2 kvm,vfio_pci List of all IOMMUs … IOMMU Group 0 00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge [1002:5a14] (rev 02) IOMMU Group 1 00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0) [1002:5a16] IOMMU Group 10 00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383] (rev 40) IOMMU Group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40) IOMMU Group 12 00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40) IOMMU Group 13 00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0…

Now with my new computer all NICs and some other stuff are in group 0.

[~] lsiommu.sh
IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 0 01:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
IOMMU Group 0 01:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c8] (rev 01)
IOMMU Group 0 01:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c6] (rev 01)
IOMMU Group 0 02:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 0 02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 0 02:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 0 02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 0 02:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 0 02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43c7] (rev 01)
IOMMU Group 0 05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 0 06:00.0 Network controller [0280]: Realtek Semiconductor Co., Ltd. Device [10ec:b822]
IOMMU Group 0 07:00.0 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 0 07:00.1 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 0 07:00.2 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 0 07:00.3 Ethernet controller [0200]: Intel Corporation 82580 Gigabit Network Connection [8086:150e] (rev 01)
IOMMU Group 0 08:00.0 USB controller [0c03]: ASMedia Technology Inc. Device [1b21:2142]
IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 00:03.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:1453]
IOMMU Group 2 09:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] [10de:1b06] (rev a1)
IOMMU Group 2 09:00.1 Audio device [0403]: NVIDIA Corporation GP102 HDMI Audio Controller [10de:10ef] (rev a1)
IOMMU Group 2 0a:00.0 VGA compatible controller [0300]: NVIDIA Corporation GM200 [GeForce GTX TITAN X] [10de:17c2] (rev a1)
IOMMU Group 2 0a:00.1 Audio device [0403]: NVIDIA Corporation GM200 High Definition Audio [10de:0fb0] (rev a1)
IOMMU Group 2 0b:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd Device [144d:a808]
IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 4 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:145a]
IOMMU Group 4 0c:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Device [1022:1456]
IOMMU Group 4 0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
IOMMU Group 5 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 5 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 5 0d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device [1022:1455]
IOMMU Group 5 0d:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 5 0d:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Device [1022:1457]
IOMMU Group 6 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 6 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 7 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 7 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 7 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 7 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 7 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 7 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 7 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric Device 18h Function 6 [1022:1466]
IOMMU Group 7 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]

I don’t know why and how to fix this.

I thought the GPUs are split into 8/8 pci lanes while the last slot at the bottom goes through the chipset with 4 pci lanes.
I didn’t know all the stuff (like usb, sata, and teh onboard nic) will share the 4 lanes though the chip.

Have you any advice ?

addition …

[~] systemctl status libvirtd.service
● libvirtd.service - Virtualization daemon
   Loaded: loaded (/lib/systemd/system/libvirtd.service; enabled; vendor preset: disabled)
  Drop-In: /etc/systemd/system/libvirtd.service.d
           └─00gentoo.conf
   Active: active (running) since Sun 2019-03-10 02:45:50 CET; 7min ago
     Docs: man:libvirtd(8)
           https://libvirt.org
 Main PID: 4354 (libvirtd)
   Memory: 33.9M
   CGroup: /system.slice/libvirtd.service
           └─4354 /usr/sbin/libvirtd

Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.973+0000: 4388: error : virGetUserID:1041 : Ungültiges Argument: Analyse des Benutzers 'tss' fehlgeschlagen
Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.973+0000: 4388: error : virGetGroupID:1124 : Ungültiges Argument: Analyse der Gruppe 'tss' fehlgeschlagen
Mär 10 02:46:04 mypc libvirtd[4354]: 2019-03-10 01:46:04.621+0000: 4373: error : virConnectNetworkEventRegisterAny:1078 : Diese Funktion wird vom Verbindungstreiber nicht unterstützt: virConnectNetworkEventRegisterAny
Mär 10 02:46:04 mypc libvirtd[4354]: 2019-03-10 01:46:04.623+0000: 4374: error : virConnectNumOfNetworks:145 : Diese Funktion wird vom Verbindungstreiber nicht unterstützt: virConnectNumOfNetworks
Mär 10 02:46:25 mypc libvirtd[4354]: 2019-03-10 01:46:25.912+0000: 4354: error : qemuMonitorIORead:609 : Kann nicht vom Monitor lesen: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
Mär 10 02:46:25 mypc libvirtd[4354]: 2019-03-10 01:46:25.912+0000: 4354: error : qemuProcessReportLogError:1933 : Interner Fehler: qemu unexpectedly closed the monitor: 2019-03-10T01:46:25.725842Z qemu-system-x86_64: -device vfio-pci,host=07:>
Mär 10 02:46:25 mypc libvirtd[4354]: Please ensure all devices within the iommu_group are bound to their vfio bus driver.
Mär 10 02:47:06 mypc libvirtd[4354]: 2019-03-10 01:47:06.200+0000: 4354: error : qemuMonitorIORead:609 : Kann nicht vom Monitor lesen: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
Mär 10 02:47:06 mypc libvirtd[4354]: 2019-03-10 01:47:06.200+0000: 4354: error : qemuProcessReportLogError:1933 : Interner Fehler: qemu unexpectedly closed the monitor: 2019-03-10T01:47:06.007792Z qemu-system-x86_64: -device vfio-pci,host=07:>
Mär 10 02:47:06 mypc libvirtd[4354]: Please ensure all devices within the iommu_group are bound to their vfio bus driver.



[~, 00h:01m:43s] journalctl -b -u libvirtd.service
-- Logs begin at Tue 2019-02-12 02:01:54 CET, end at Sun 2019-03-10 02:51:36 CET. --
Mär 10 02:45:50 mypc systemd[1]: Starting Virtualization daemon...
Mär 10 02:45:50 mypc systemd[1]: Started Virtualization daemon.
Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.967+0000: 4388: info : libvirt version: 4.9.0
Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.967+0000: 4388: info : hostname: mypc
Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.967+0000: 4388: error : virFirewallValidateBackend:191 : direct firewall backend requested, but /sbin/ebtables is not available: Datei oder Verzeichnis nicht gefunden
Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.967+0000: 4388: error : virFirewallApply:902 : Interner Fehler: Failed to initialize a valid firewall backend
Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.973+0000: 4388: error : virGetUserID:1041 : Ungültiges Argument: Analyse des Benutzers 'tss' fehlgeschlagen
Mär 10 02:45:50 mypc libvirtd[4354]: 2019-03-10 01:45:50.973+0000: 4388: error : virGetGroupID:1124 : Ungültiges Argument: Analyse der Gruppe 'tss' fehlgeschlagen
Mär 10 02:46:04 mypc libvirtd[4354]: 2019-03-10 01:46:04.621+0000: 4373: error : virConnectNetworkEventRegisterAny:1078 : Diese Funktion wird vom Verbindungstreiber nicht unterstützt: virConnectNetworkEventRegisterAny
Mär 10 02:46:04 mypc libvirtd[4354]: 2019-03-10 01:46:04.623+0000: 4374: error : virConnectNumOfNetworks:145 : Diese Funktion wird vom Verbindungstreiber nicht unterstützt: virConnectNumOfNetworks
Mär 10 02:46:25 mypc libvirtd[4354]: 2019-03-10 01:46:25.912+0000: 4354: error : qemuMonitorIORead:609 : Kann nicht vom Monitor lesen: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
Mär 10 02:46:25 mypc libvirtd[4354]: 2019-03-10 01:46:25.912+0000: 4354: error : qemuProcessReportLogError:1933 : Interner Fehler: qemu unexpectedly closed the monitor: 2019-03-10T01:46:25.725842Z qemu-system-x86_64: -device vfio-pci,host=07:>
Mär 10 02:46:25 mypc libvirtd[4354]: Please ensure all devices within the iommu_group are bound to their vfio bus driver.
Mär 10 02:47:06 mypc libvirtd[4354]: 2019-03-10 01:47:06.200+0000: 4354: error : qemuMonitorIORead:609 : Kann nicht vom Monitor lesen: Die Verbindung wurde vom Kommunikationspartner zurückgesetzt
Mär 10 02:47:06 mypc libvirtd[4354]: 2019-03-10 01:47:06.200+0000: 4354: error : qemuProcessReportLogError:1933 : Interner Fehler: qemu unexpectedly closed the monitor: 2019-03-10T01:47:06.007792Z qemu-system-x86_64: -device vfio-pci,host=07:>
Mär 10 02:47:06 mypc libvirtd[4354]: Please ensure all devices within the iommu_group are bound to their vfio bus driver.

somehow it looks like I need a bigger mobo with more pci lanes
or I have to build four bridges and hope I can access the VMs via network and do some nmap, tcpdump, and install pfsense behind the bridges.
It would be nice if you can confirm this, so i can uses bridges.

will I need a x399/mobo to get my nics out of group 0 ?
or are there some other options to choose from beside of TR.

Holla, if you need more info.