[Proxmox] nic passthrough fails

Hello,

Like the titles says, i have problems with my nic passthrough.
my system is a Dell 760 2x X5650 ram 96GiB
I see the iommu groups and when i edit the vm’s config file
the passthroughshows in the list.
But when i start the vm i get this error

I’m lost what am i doing wrong, please some help…i am a novice/newby/little grasshopper

kvm: -device vfio-pci,host=07:00.1,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: vfio error: 0000:07:00.1: failed to setup container for group 17: failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: command ‘/usr/bin/kvm -id 106 -name Pfsense -chardev ‘socket,id=qmp,path=/var/run/qemu-server/106.qmp,server,nowait’ -mon ‘chardev=qmp,mode=control’ -pidfile /var/run/qemu-server/106.pid -daemonize -smbios ‘type=1,uuid=561af69e-78f3-4f75-8a95-3c0e5f63cf05’ -smp ‘2,sockets=1,cores=2,maxcpus=2’ -nodefaults -boot ‘menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg’ -vga std -vnc unix:/var/run/qemu-server/106.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -readconfig /usr/share/qemu-server/pve-q35.cfg -device ‘usb-tablet,id=tablet,bus=ehci.0,port=1’ -device ‘vfio-pci,host=07:00.1,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on’ -device ‘vfio-pci,host=07:00.0,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1’ -device ‘virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3’ -iscsi ‘initiator-name=iqn.1993-08.org.debian:01:d6faace9ae8’ -drive ‘if=none,id=drive-ide2,media=cdrom,aio=threads’ -device ‘ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100’ -device ‘virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5’ -drive ‘file=/mnt/ssd/images/106/vm-106-disk-1.qcow2,if=none,id=drive-scsi0,cache=writethrough,format=qcow2,aio=threads,detect-zeroes=on’ -device ‘scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200’ -machine ‘type=q35’’ failed: exit code 1

/qemu-server# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/17/devices/0000:07:00.0
/sys/kernel/iommu_groups/17/devices/0000:07:00.1
/sys/kernel/iommu_groups/17/devices/0000:06:02.0
/sys/kernel/iommu_groups/7/devices/0000:00:09.0
/sys/kernel/iommu_groups/25/devices/0000:fe:04.2
/sys/kernel/iommu_groups/25/devices/0000:fe:04.0
/sys/kernel/iommu_groups/25/devices/0000:fe:04.3
/sys/kernel/iommu_groups/25/devices/0000:fe:04.1
/sys/kernel/iommu_groups/15/devices/0000:03:00.0
/sys/kernel/iommu_groups/33/devices/0000:ff:06.2
/sys/kernel/iommu_groups/33/devices/0000:ff:06.0
/sys/kernel/iommu_groups/33/devices/0000:ff:06.3
/sys/kernel/iommu_groups/33/devices/0000:ff:06.1
/sys/kernel/iommu_groups/5/devices/0000:00:06.0
/sys/kernel/iommu_groups/23/devices/0000:fe:02.5
/sys/kernel/iommu_groups/23/devices/0000:fe:02.3
/sys/kernel/iommu_groups/23/devices/0000:fe:02.1
/sys/kernel/iommu_groups/23/devices/0000:fe:02.4
/sys/kernel/iommu_groups/23/devices/0000:fe:02.2
/sys/kernel/iommu_groups/23/devices/0000:fe:02.0
/sys/kernel/iommu_groups/13/devices/0000:01:00.1
/sys/kernel/iommu_groups/13/devices/0000:01:00.0
/sys/kernel/iommu_groups/31/devices/0000:ff:04.0
/sys/kernel/iommu_groups/31/devices/0000:ff:04.3
/sys/kernel/iommu_groups/31/devices/0000:ff:04.1
/sys/kernel/iommu_groups/31/devices/0000:ff:04.2
/sys/kernel/iommu_groups/3/devices/0000:00:04.0
/sys/kernel/iommu_groups/21/devices/0000:0a:04.0
/sys/kernel/iommu_groups/21/devices/0000:0d:00.1
/sys/kernel/iommu_groups/21/devices/0000:0d:00.0
/sys/kernel/iommu_groups/11/devices/0000:00:1e.0
/sys/kernel/iommu_groups/11/devices/0000:10:03.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/28/devices/0000:ff:00.0
/sys/kernel/iommu_groups/28/devices/0000:ff:00.1
/sys/kernel/iommu_groups/18/devices/0000:08:00.1
/sys/kernel/iommu_groups/18/devices/0000:08:00.0
/sys/kernel/iommu_groups/18/devices/0000:06:04.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.2
/sys/kernel/iommu_groups/8/devices/0000:00:14.0
/sys/kernel/iommu_groups/8/devices/0000:00:14.1
/sys/kernel/iommu_groups/26/devices/0000:fe:05.3
/sys/kernel/iommu_groups/26/devices/0000:fe:05.1
/sys/kernel/iommu_groups/26/devices/0000:fe:05.2
/sys/kernel/iommu_groups/26/devices/0000:fe:05.0
/sys/kernel/iommu_groups/16/devices/0000:05:00.0
/sys/kernel/iommu_groups/6/devices/0000:00:07.0
/sys/kernel/iommu_groups/24/devices/0000:fe:03.4
/sys/kernel/iommu_groups/24/devices/0000:fe:03.2
/sys/kernel/iommu_groups/24/devices/0000:fe:03.0
/sys/kernel/iommu_groups/24/devices/0000:fe:03.1
/sys/kernel/iommu_groups/14/devices/0000:02:00.1
/sys/kernel/iommu_groups/14/devices/0000:02:00.0
/sys/kernel/iommu_groups/32/devices/0000:ff:05.3
/sys/kernel/iommu_groups/32/devices/0000:ff:05.1
/sys/kernel/iommu_groups/32/devices/0000:ff:05.2
/sys/kernel/iommu_groups/32/devices/0000:ff:05.0
/sys/kernel/iommu_groups/4/devices/0000:00:05.0
/sys/kernel/iommu_groups/22/devices/0000:fe:00.0
/sys/kernel/iommu_groups/22/devices/0000:fe:00.1
/sys/kernel/iommu_groups/12/devices/0000:00:1f.2
/sys/kernel/iommu_groups/12/devices/0000:00:1f.0
/sys/kernel/iommu_groups/30/devices/0000:ff:03.4
/sys/kernel/iommu_groups/30/devices/0000:ff:03.2
/sys/kernel/iommu_groups/30/devices/0000:ff:03.0
/sys/kernel/iommu_groups/30/devices/0000:ff:03.1
/sys/kernel/iommu_groups/2/devices/0000:00:03.0
/sys/kernel/iommu_groups/20/devices/0000:0b:00.0
/sys/kernel/iommu_groups/20/devices/0000:0b:00.1
/sys/kernel/iommu_groups/20/devices/0000:0a:02.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.0
/sys/kernel/iommu_groups/10/devices/0000:00:1d.7
/sys/kernel/iommu_groups/10/devices/0000:00:1d.1
/sys/kernel/iommu_groups/29/devices/0000:ff:02.3
/sys/kernel/iommu_groups/29/devices/0000:ff:02.1
/sys/kernel/iommu_groups/29/devices/0000:ff:02.4
/sys/kernel/iommu_groups/29/devices/0000:ff:02.2
/sys/kernel/iommu_groups/29/devices/0000:ff:02.0
/sys/kernel/iommu_groups/29/devices/0000:ff:02.5
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/19/devices/0000:09:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1a.0
/sys/kernel/iommu_groups/9/devices/0000:00:1a.7
/sys/kernel/iommu_groups/9/devices/0000:00:1a.1
/sys/kernel/iommu_groups/27/devices/0000:fe:06.1
/sys/kernel/iommu_groups/27/devices/0000:fe:06.2
/sys/kernel/iommu_groups/27/devices/0000:fe:06.0
/sys/kernel/iommu_groups/27/devices/0000:fe:06.3

failed to setup container for group 17: failed to set iommu for container: Operation not permitted

You must pass through everything in IOMMU group 17. If you run the IOMMU script:

#!/bin/bash
shopt -s nullglob
for d in /sys/kernel/iommu_groups/*/devices/*; do
    n=${d#*/iommu_groups/*}; n=${n%%/*}
    printf 'IOMMU Group %s ' "$n"
    lspci -nns "${d##*/}"
done;

It will tell you exactly what’s in group 17. If you want to avoid this, you can use the ACS Override Patch to bypass it, but that’s potentially dangerous and I would recommend trying to avoid it. You’ll also need to compile the kernel from source to do this, or find a 3rd party binary distribution of the kernel.

1 Like

I ran the script and this is the output for group 17.

IOMMU Group 17 06:02.0 PCI bridge [0604]: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch [111d:8061] (rev 01)
IOMMU Group 17 07:00.0 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)
IOMMU Group 17 07:00.1 Ethernet controller [0200]: Broadcom Limited NetXtreme II BCM5709 Gigabit Ethernet [14e4:1639] (rev 20)

The PCI bridge, is this part of the nic?

My VM config file:

boot: dcn
bootdisk: scsi0
cores: 2
ide2: none,media=cdrom
machine: q35
hostpci0: 06:02.0;07:00.1;07:00.0,pcie=1
memory: 4096
name: Pfsense
numa: 0
ostype: l26
scsi0: virtual_ssd:106/vm-106-disk-1.qcow2,cache=writethrough,size=32G
scsihw: virtio-scsi-pci
smbios1: uuid=561af69e-78f3-4f75-8a95-3c0e5f63cf05
sockets: 1

stil will not boot! grrrr

Same error?

Ehh, hard to tell. Could be part of the nic, could be part of the motherboard. Hard to say. Judging by the manufacturer though, I’d argue nic.

I think it’s the same error

kvm: -device vfio-pci,host=06:02.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on: vfio error: 0000:06:02.0: failed to setup container for group 17: failed to set iommu for container: Operation not permitted
TASK ERROR: start failed: command ‘/usr/bin/kvm -id 106 -name Pfsense -chardev ‘socket,id=qmp,path=/var/run/qemu-server/106.qmp,server,nowait’ -mon ‘chardev=qmp,mode=control’ -pidfile /var/run/qemu-server/106.pid -daemonize -smbios ‘type=1,uuid=561af69e-78f3-4f75-8a95-3c0e5f63cf05’ -smp ‘2,sockets=1,cores=2,maxcpus=2’ -nodefaults -boot ‘menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg’ -vga std -vnc unix:/var/run/qemu-server/106.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -readconfig /usr/share/qemu-server/pve-q35.cfg -device ‘usb-tablet,id=tablet,bus=ehci.0,port=1’ -device ‘vfio-pci,host=06:02.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on’ -device ‘vfio-pci,host=07:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1’ -device ‘vfio-pci,host=07:00.0,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2’ -device ‘virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3’ -iscsi ‘initiator-name=iqn.1993-08.org.debian:01:d6faace9ae8’ -drive ‘if=none,id=drive-ide2,media=cdrom,aio=threads’ -device ‘ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100’ -device ‘virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5’ -drive ‘file=/mnt/ssd/images/106/vm-106-disk-1.qcow2,if=none,id=drive-scsi0,cache=writethrough,format=qcow2,aio=threads,detect-zeroes=on’ -device ‘scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200’ -machine ‘type=q35’’ failed: exit code 1

This was in the syslog:

Aug 14 21:45:12 virtual pmxcfs[2048]: [status] notice: received log
Aug 14 21:45:19 virtual pvedaemon[2158]: vm 106 - unable to parse value of ‘hostpci0’ - format error
host: value does not match the regex pattern
Aug 14 21:45:19 virtual pveproxy[17362]: vm 106 - unable to parse value of ‘hostpci0’ - format error
host: value does not match the regex pattern
Aug 14 21:45:58 virtual pvedaemon[26009]: start VM 106: UPID:virtual:00006599:0013EC31:5B733176:qmstart:106:root@pam:
Aug 14 21:45:58 virtual pvedaemon[16538]: root@pam starting task UPID:virtual:00006599:0013EC31:5B733176:qmstart:106:root@pam:
Aug 14 21:45:58 virtual systemd[1]: Started 106.scope.
Aug 14 21:45:58 virtual kernel: vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param “allow_unsafe_interrupts” to enable VFIO IOMMU support on this platform
Aug 14 21:45:58 virtual pvedaemon[26009]: start failed: command ‘/usr/bin/kvm -id 106 -name Pfsense -chardev ‘socket,id=qmp,path=/var/run/qemu-server/106.qmp,server,nowait’ -mon ‘chardev=qmp,mode=control’ -pidfile /var/run/qemu-server/106.pid -daemonize -smbios ‘type=1,uuid=561af69e-78f3-4f75-8a95-3c0e5f63cf05’ -smp ‘2,sockets=1,cores=2,maxcpus=2’ -nodefaults -boot ‘menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg’ -vga std -vnc unix:/var/run/qemu-server/106.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -readconfig /usr/share/qemu-server/pve-q35.cfg -device ‘usb-tablet,id=tablet,bus=ehci.0,port=1’ -device ‘vfio-pci,host=06:02.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on’ -device ‘vfio-pci,host=07:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1’ -device ‘vfio-pci,host=07:00.0,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2’ -device ‘virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3’ -iscsi ‘initiator-name=iqn.1993-08.org.debian:01:d6faace9ae8’ -drive ‘if=none,id=drive-ide2,media=cdrom,aio=threads’ -device ‘ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100’ -device ‘virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5’ -drive ‘file=/mnt/ssd/images/106/vm-106-disk-1.qcow2,if=none,id=drive-scsi0,cache=writethrough,format=qcow2,aio=threads,detect-zeroes=on’ -device ‘scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200’ -machine ‘type=q35’’ failed: exit code 1
Aug 14 21:45:58 virtual pvedaemon[16538]: root@pam end task UPID:virtual:00006599:0013EC31:5B733176:qmstart:106:root@pam: start failed: command ‘/usr/bin/kvm -id 106 -name Pfsense -chardev ‘socket,id=qmp,path=/var/run/qemu-server/106.qmp,server,nowait’ -mon ‘chardev=qmp,mode=control’ -pidfile /var/run/qemu-server/106.pid -daemonize -smbios ‘type=1,uuid=561af69e-78f3-4f75-8a95-3c0e5f63cf05’ -smp ‘2,sockets=1,cores=2,maxcpus=2’ -nodefaults -boot ‘menu=on,strict=on,reboot-timeout=1000,splash=/usr/share/qemu-server/bootsplash.jpg’ -vga std -vnc unix:/var/run/qemu-server/106.vnc,x509,password -cpu kvm64,+lahf_lm,+sep,+kvm_pv_unhalt,+kvm_pv_eoi,enforce -m 4096 -readconfig /usr/share/qemu-server/pve-q35.cfg -device ‘usb-tablet,id=tablet,bus=ehci.0,port=1’ -device ‘vfio-pci,host=06:02.0,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,multifunction=on’ -device ‘vfio-pci,host=07:00.1,id=hostpci0.1,bus=ich9-pcie-port-1,addr=0x0.1’ -device ‘vfio-pci,host=07:00.0,id=hostpci0.2,bus=ich9-pcie-port-1,addr=0x0.2’ -device ‘virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3’ -iscsi ‘initiator-name=iqn.1993-08.org.debian:01:d6faace9ae8’ -drive ‘if=none,id=drive-ide2,media=cdrom,aio=threads’ -device ‘ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=100’ -device ‘virtio-scsi-pci,id=scsihw0,bus=pci.0,addr=0x5’ -drive ‘file=/mnt/ssd/images/106/vm-106-disk-1.qcow2,if=none,id=drive-scsi0,cache=writethrough,format=qcow2,aio=threads,detect-zeroes=on’ -device ‘scsi-hd,bus=scsihw0.0,channel=0,scsi-id=0,lun=0,drive=drive-scsi0,id=scsi0,bootindex=200’ -machine ‘type=q35’’ failed: exit code 1
Aug 14 21:46:00 virtual systemd[1]: Starting Proxmox VE replication runner…
Aug 14 21:46:01 virtual systemd[1]: Started Proxmox VE replication runner.

Let’s check a couple things here. I’ll need the outputs of the following:

cat /etc/default/grub
dmesg | grep -e vfio
cat /etc/modprobe.d/vfio.conf

Also, I think you’ll need to reconfigure your vm config file like so:

hostpci0: 06:02.0,pcie=1
hostpci1: 07:00.0,pcie=1
hostpci2: 07:00.1,pcie=1

cat /etc/default/grub

# If you change this file, run 'update-grub' afterwards to update
# /boot/grub/grub.cfg.
# For full documentation of the options in this file, see:
#   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
GRUB_CMDLINE_LINUX=""

# Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

# Uncomment to enable BadRAM filtering, modify to suit your needs
# This works with Linux (no patch required) and with any kernel that obtains
# the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
#GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

# Uncomment to disable graphical terminal (grub-pc only)
#GRUB_TERMINAL=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=640x480

# Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
#GRUB_DISABLE_LINUX_UUID=true

# Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

# Uncomment to get a beep at grub start
#GRUB_INIT_TUNE="480 440 1"

dmesg | grep -e vfio

[  145.373960] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[  210.527196] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[  417.499717] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[  587.362824] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 1664.774173] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7034.596352] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7063.894453] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7113.846345] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7241.561266] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7279.072419] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7292.278932] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7296.678689] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7297.721437] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 7390.397132] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[ 8095.454899] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[13056.436762] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform
[18630.694142] vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

cat /etc/modprobe.d/vfio.conf

cat: /etc/modprobe.d/vfio.conf: No such file or directory
hostpci0: 06:02.0,pcie=1
hostpci1: 07:00.0,pcie=1
hostpci2: 07:00.1,pcie=1

DIDN’T WORK

I edited your post for readability. Using 3 backticks (`) will allow you to create a code block so you can just paste in the output.


Did you follow the proxmox passthrough guide?

https://pve.proxmox.com/wiki/Pci_passthrough

If not, I recommend going back and following that.

Can you also show me your /etc/modules? (cat /etc/modules)

I did follow that guide!

cat /etc/modules

/etc/modules: kernel modules to load at boot time.

This file contains the names of kernel modules that should be loaded

at boot time, one per line. Lines beginning with “#” are ignored.

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Hmmm, okay. So you need to make sure you’ve got vfio loaded on the devices you want to pass through, otherwise it may not work.

I’ll have to try to replicate the issue when I get off work.

Super thnx for the help.

i tried and added “pcie_acs_override=downstream” to the grub file. but stil a no go

Would love to see this work :slight_smile:

Yeah got it working.
It’s a 4 port nic and it seems i have to do
hostpci1: 07:00,pcie=1
hostpci2: 08:00,pcie=1
and now the VM boots.
It looks like I have to pass-trough the entire card.

I also created /etc/modprobe.d/vfio.conf, but this is a empty file. Don’t think it is the solution.
Did some little test and it was te fact that the vfio.conf file was missing.
can now boot it with only the hostpci1

I have one more question (for the moment :wink:)
I wonder if i can read the nic naming like this.
I added a list of the nic’s in the server seen by proxmox.
the enp11s0fx does this translate to

11:00.x
hostpci1: 11:00.0,pcie=1
hostpci2: 11:00.1,pcie=1

/sys/kernel/iommu_groups/11/devices/0000:00:1e.0
/sys/kernel/iommu_groups/11/devices/0000:10:03.0

nics

To be honest, I don’t know. My recommendation is to run the following:

 udevadm info -e | grep -A 9 ^P.*enp11*

and see if that helps. udevadm info can be used to get information regarding udev device naming (which is the tool responsible for assigning names to all devices)