Proxmox homelab & IOMMU troubleshooting

Software & Operating Systems Linux
1

So I was using ESXi for a long time on my humble server to run VM's for testing and services for me and my friends. When I made the switch from Windows to Arch on my main PC (ASUS G75VX) I was no longer able to use the vSphere client. That was the reason enough for me to try out an alternative, Proxmox.

My server has an PCIe dual NIC card and I want to use IOMMU to pass it to an PFsense VM. With help of Proxmox wiki I was able to turn on IOMMU and followed the configuration required to make it work.

I haven't been successful, yet, because I only got one NIC to appear in my PFsense VM and it did not work. I gather all the information that I think is relevant for this topic below. If anybody has experience with this or knows how to help me with the troubleshooting it would be greatly appreciated.

My thoughts

The onboard NIC and the PCIe dual NIC card both use RTL8111/8168/8411, I saw alot of people blacklisting the driver of their PCIe cards to make IOMMU work. Is this a problem in my case or is the blacklisting of drivers only use for graphics cards?

In lshw the Dual NIC is shown under /0/100/9/ but under lspci -nnk their 07:00.0 and 09:00.0 and people use the ID form lspci to pass it through to the VM like so:
hostpci0: 07:00.0,pcie=1,driver=vfio
hostpci0: 09:00.0,pcie=1,driver=vfio
But shouldn't it be 1 ID's because there is 1 PCIe dual NIC card?

Details of the Proxmox server:

Motherboard: ASUS M5A97 R2.0
CPU: AMD FX-8320E

~# lshw -short
H/W path        Device     Class       Description
==================================================
                           system      Computer
/0                         bus         Motherboard
/0/0                       memory      15GiB System memory
/0/1                       processor   AMD FX-8320E Eight-Core Processor
/0/100                     bridge      RD9x0/RX980 Host Bridge
/0/100/0.2                 generic     RD890S/RD990 I/O Memory Management Unit (IOMMU)
/0/100/2                   bridge      RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0)
/0/100/2/0                 display     Cedar [Radeon HD 5000/6000/7350/8350 Series]
/0/100/2/0.1               multimedia  Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series]
/0/100/4                   bridge      RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 0)
/0/100/4/0      eth0       network     RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
/0/100/5                   bridge      RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 1)
/0/100/5/0                 bus         ASM1042 SuperSpeed USB Host Controller
/0/100/5/0/0    usb9       bus         xHCI Host Controller
/0/100/5/0/1    usb8       bus         xHCI Host Controller
/0/100/7                   bridge      RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 3)
/0/100/7/0                 bus         ASM1042 SuperSpeed USB Host Controller
/0/100/7/0/0    usb11      bus         xHCI Host Controller
/0/100/7/0/1    usb10      bus         xHCI Host Controller
/0/100/9                   bridge      RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 4)
/0/100/9/0                 bridge      Pericom Semiconductor
/0/100/9/0/1               bridge      Pericom Semiconductor
/0/100/9/0/1/0  eth1       network     RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
/0/100/9/0/2               bridge      Pericom Semiconductor
/0/100/9/0/3               bridge      Pericom Semiconductor
/0/100/9/0/3/0  eth2       network     RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
/0/100/11                  storage     SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]
/0/100/12                  bus         SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/12/1     usb4       bus         OHCI PCI host controller
/0/100/12.2                bus         SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/12.2/1   usb1       bus         EHCI Host Controller
/0/100/13                  bus         SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/13/1     usb5       bus         OHCI PCI host controller
/0/100/13.2                bus         SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/13.2/1   usb2       bus         EHCI Host Controller
/0/100/14                  bus         SBx00 SMBus Controller
/0/100/14.2                multimedia  SBx00 Azalia (Intel HDA)
/0/100/14.3                bridge      SB7x0/SB8x0/SB9x0 LPC host controller
/0/100/14.4                bridge      SBx00 PCI to PCI Bridge
/0/100/14.5                bus         SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
/0/100/14.5/1   usb6       bus         OHCI PCI host controller
/0/100/16                  bus         SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
/0/100/16/1     usb7       bus         OHCI PCI host controller
/0/100/16.2                bus         SB7x0/SB8x0/SB9x0 USB EHCI Controller
/0/100/16.2/1   usb3       bus         EHCI Host Controller
/0/101                     bridge      Family 15h Processor Function 0
/0/102                     bridge      Family 15h Processor Function 1
/0/103                     bridge      Family 15h Processor Function 2
/0/104                     bridge      Family 15h Processor Function 3
/0/105                     bridge      Family 15h Processor Function 4
/0/106                     bridge      Family 15h Processor Function 5
/0/2            scsi0      storage     
/0/2/0.0.0      /dev/sda   disk        64GB ADATA SP900
/0/2/0.0.0/1    /dev/sda1  volume      1023KiB BIOS Boot partition
/0/2/0.0.0/2    /dev/sda2  volume      255MiB Windows FAT volume
/0/2/0.0.0/3    /dev/sda3  volume      59GiB LVM Physical Volume
/0/3            scsi2      storage     
/0/3/0.0.0      /dev/sdb   disk        640GB ST3640623AS
/0/3/0.0.0/1    /dev/sdb1  volume      596GiB EFI partition


~# dmesg | grep AMD-Vi
[    1.360374] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.360375] AMD-Vi: Interrupt remapping enabled
[    1.360484] AMD-Vi: Lazy IO/TLB flushing enabled

~# lsmod | grep iommu
amd_iommu_v2           20480  1 amdkfd
vfio_iommu_type1       20480  0 
vfio                   28672  2 vfio_iommu_type1,vfio_pci

~# cat /etc/pve/qemu-server/100.conf
bootdisk: sata0
cores: 1
hostpci0: 07:00.0,pcie=1,driver=vfio
hostpci0: 09:00.0,pcie=1,driver=vfio
ide2: none,media=cdrom
machine: q35
memory: 1024
name: pfs
numa: 0
ostype: other
sata0: local-lvm:vm-100-disk-1,cache=writeback,size=20G
scsihw: virtio-scsi-pci
smbios1: uuid=0c35a25f-5a07-4318-996e-8552672e70f3
sockets: 1

~# cat /etc/default/grub 
 # If you change this file, run 'update-grub' afterwards to update
 # /boot/grub/grub.cfg.
 # For full documentation of the options in this file, see:
 #   info -f grub -n 'Simple configuration'

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Proxmox Virtual Environment"
GRUB_CMDLINE_LINUX_DEFAULT="quiet amd_iommu=on iommu=pt iommu=1"
GRUB_CMDLINE_LINUX=""

 # Disable os-prober, it might add menu entries for each guest
GRUB_DISABLE_OS_PROBER=true

 # Uncomment to enable BadRAM filtering, modify to suit your needs
 # This works with Linux (no patch required) and with any kernel that obtains
 # the memory map information from GRUB (GNU Mach, kernel of FreeBSD ...)
 #GRUB_BADRAM="0x01234567,0xfefefefe,0x89abcdef,0xefefefef"

 # Uncomment to disable graphical terminal (grub-pc only)
 #GRUB_TERMINAL=console

 # The resolution used on graphical terminal
 # note that you can use only modes which your graphic card supports via VBE
 # you can see them in real GRUB with the command `vbeinfo'
 #GRUB_GFXMODE=640x480

 # Uncomment if you don't want GRUB to pass "root=UUID=xxx" parameter to Linux
 #GRUB_DISABLE_LINUX_UUID=true

 # Disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY="true"

 # Uncomment to get a beep at grub start
 #GRUB_INIT_TUNE="480 440 1"


~# find /sys/kernel/iommu_groups/ -type l
/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:02.0
/sys/kernel/iommu_groups/2/devices/0000:00:04.0
/sys/kernel/iommu_groups/3/devices/0000:00:05.0
/sys/kernel/iommu_groups/4/devices/0000:00:07.0
/sys/kernel/iommu_groups/5/devices/0000:00:09.0
/sys/kernel/iommu_groups/6/devices/0000:00:11.0
/sys/kernel/iommu_groups/7/devices/0000:00:12.0
/sys/kernel/iommu_groups/7/devices/0000:00:12.2
/sys/kernel/iommu_groups/8/devices/0000:00:13.0
/sys/kernel/iommu_groups/8/devices/0000:00:13.2
/sys/kernel/iommu_groups/9/devices/0000:00:14.0
/sys/kernel/iommu_groups/10/devices/0000:00:14.2
/sys/kernel/iommu_groups/11/devices/0000:00:14.3
/sys/kernel/iommu_groups/12/devices/0000:00:14.4
/sys/kernel/iommu_groups/13/devices/0000:00:14.5
/sys/kernel/iommu_groups/14/devices/0000:00:16.0
/sys/kernel/iommu_groups/14/devices/0000:00:16.2
/sys/kernel/iommu_groups/15/devices/0000:01:00.0
/sys/kernel/iommu_groups/15/devices/0000:01:00.1
/sys/kernel/iommu_groups/16/devices/0000:02:00.0
/sys/kernel/iommu_groups/17/devices/0000:03:00.0
/sys/kernel/iommu_groups/18/devices/0000:04:00.0
/sys/kernel/iommu_groups/19/devices/0000:05:00.0
/sys/kernel/iommu_groups/20/devices/0000:06:01.0
/sys/kernel/iommu_groups/21/devices/0000:06:02.0
/sys/kernel/iommu_groups/22/devices/0000:06:03.0
/sys/kernel/iommu_groups/23/devices/0000:07:00.0
/sys/kernel/iommu_groups/24/devices/0000:09:00.0


~# lspci -nnk
00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge [1002:5a14] (rev 02)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge [1002:5a14]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RD890S/RD990 I/O Memory Management Unit (IOMMU) [1002:5a23]
00:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0) [1002:5a16]
	Kernel driver in use: pcieport
00:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 0) [1002:5a18]
	Kernel driver in use: pcieport
00:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 1) [1002:5a19]
	Kernel driver in use: pcieport
00:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 3) [1002:5a1b]
	Kernel driver in use: pcieport
00:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GPP Port 4) [1002:5a1c]
	Kernel driver in use: pcieport
00:11.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] [1002:4391] (rev 40)
	Subsystem: ASUSTeK Computer Inc. Device [1043:84dd]
	Kernel driver in use: ahci
00:12.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
	Kernel driver in use: ohci-pci
00:12.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
	Kernel driver in use: ehci-pci
00:13.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
	Kernel driver in use: ohci-pci
00:13.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
	Kernel driver in use: ehci-pci
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385] (rev 42)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller [1002:4385]
	Kernel driver in use: piix4_smbus
00:14.2 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) [1002:4383] (rev 40)
	Subsystem: ASUSTeK Computer Inc. Device [1043:8444]
	Kernel driver in use: snd_hda_intel
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d] (rev 40)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller [1002:439d]
00:14.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge [1002:4384] (rev 40)
00:14.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller [1002:4399]
	Kernel driver in use: ohci-pci
00:16.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4397]
	Kernel driver in use: ohci-pci
00:16.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller [1002:4396]
	Kernel driver in use: ehci-pci
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0 [1022:1600]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1 [1022:1601]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2 [1022:1602]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3 [1022:1603]
	Kernel driver in use: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4 [1022:1604]
	Kernel driver in use: fam15h_power
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5 [1022:1605]
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] [1002:68f9]
	Subsystem: ASUSTeK Computer Inc. Device [1043:04b3]
	Kernel driver in use: radeon
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300/7300 Series] [1002:aa68]
	Subsystem: ASUSTeK Computer Inc. Device [1043:aa68]
	Kernel driver in use: snd_hda_intel
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 09)
	Subsystem: ASUSTeK Computer Inc. P8 series motherboard [1043:8505]
	Kernel driver in use: r8169
03:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042]
	Subsystem: ASUSTeK Computer Inc. P8B WS Motherboard [1043:8488]
	Kernel driver in use: xhci_hcd
04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042]
	Subsystem: ASUSTeK Computer Inc. P8B WS Motherboard [1043:8488]
	Kernel driver in use: xhci_hcd
05:00.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:2404] (rev 05)
	Kernel driver in use: pcieport
06:01.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:2404] (rev 05)
	Kernel driver in use: pcieport
06:02.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:2404] (rev 05)
	Kernel driver in use: pcieport
06:03.0 PCI bridge [0604]: Pericom Semiconductor Device [12d8:2404] (rev 05)
	Kernel driver in use: pcieport
07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 07)
	Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
	Kernel driver in use: r8169
09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 07)
	Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
	Kernel driver in use: r8169

If things are unclear or non complete I will happily explain more or get more details form the server.

Thanks in advance,
FUBARnl

edit: formatting, words, code blocks

2

Just a piece of advice. Three backticks (`) can create code blocks, which would make it easier for us to read the details within the details tag. Also, thanks for using the details tag!

So, this realtek adapter is at 07:00.0 and 09:00.0, so you'll have to pass through both of them.

Try this: 

hostpci0: 07:00.0,pcie=1,driver=vfio
hostpci1: 09:00.0,pcie=1,driver=vfio

If I understand the Proxmox config files properly, your original config was trying to assign two devices to the same virtual pci port.

You'll also have to bind the vfio-pci driver to those nic's, otherwise the VM may crash.

3

Thanks for the code block syntax, edited it immediately, much better.
I had not figured out what those variables stood for and didn't find any documantation, but it sounds really logical. I will try it out in a few hours and i do belive I already loaded those drivers in the PFsense VM but I will double check!

Thanks for the quick response/help :slight_smile:

1 Like
4

Changed the config but PFsense finds no ports, also tried with pciconf command but the NIC's are showing. I'm also wonder if it has something to do with loading a stub driver for the PCIe card on the host(Proxmox).

5

Hmm, I'm not experienced with proxmox passthrough. I'm going to read up on it this weekend and I'll get back to you as soon as I get something that may help. Sorry I don't have an immediate answer.

6

Just as an aside because you said you could no longer use EXSI due to not being able to use the vSphere Client, if your running version 6 or higher (I think its 6 but regardless) then there is a web client that can be accessed via the server's IP address. Its not my favorite web client, but its functional and you can do everything you could do in the old Client, just some of the stuff is now in different places.

1 Like
7

Real Quick :slight_smile:

8

Yes that's very true and ESXI works wonderfully! But I prefer free and open software, so that's another reason for me to switch to Proxmox or any other open and free solution. But thanks for the insight.

2 Likes
9

So, i've been trying but no succes. I saw in a guide that they used the pcie var as ID but no succes.

hostpci0: 07:00.0,pcie=1,driver=vfio
hostpci0: 09:00.0,pcie=2,driver=vfio

Have been asking around but nobody has experience with this.

10

So, I think you're going to need to edit the module configuration /etc/modprobe.d to set the realtek nic's to use vfio-pci as the default module.

From there we can be sure they're not being messed with by the host OS.

11

This is how i have it configured:

root@vms:/etc/modprobe.d# cat vfio.conf 
options vfio-pci ids=10ec:0123

And the other configs I created to enable unsafe interrupts:

root@vms:/etc/modprobe.d# cat iommu_unsafe_interrupts.conf 
options vfio_iommu_type1 allow_unsafe_interrupts=1

root@vms:/etc/modprobe.d# cat kvm_iommu_map_guest.conf 
options kvm allow_unsafe_assigned_interrupts=1
12

Hmmm, that should be working.... I'm out of my depth on this one. :confused:

You may have to blacklist the realtek driver from the devices. The problem there is it appears all your NIC's are realtek. You may have to manually unbind and rebind specific devices post-boot.

This page has some useful info and is worth having a read:

https://wiki.archlinux.org/index.php/PCI_passthrough_via_OVMF

13

Thanks for the tip, arch wiki is really nice.
I found out how to manually unbind and bind drivers with these commands (in my case).

echo -n "0000:07:00.0" > /sys/bus/pci/drivers/r8169/unbind
echo -n "0000:09:00.0" > /sys/bus/pci/drivers/r8169/unbind
echo -n "0000:07:00.0" > /sys/bus/pci/drivers/vfio-pci/bind
echo -n "0000:09:00.0" > /sys/bus/pci/drivers/vfio-pci/bind

And then we can see the drivers are loaded.

07:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 07)
	Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
	Kernel driver in use: vfio-pci
09:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 07)
	Subsystem: Realtek Semiconductor Co., Ltd. Device [10ec:0123]
	Kernel driver in use: vfio-pci

When booting in to PFsense the network nics are not detected, and when I search for the virtio pci i find only one.

pciconf -l | grep virtio
virtio_pci0@pci0:6:3:0: class=0x00ff00 card=0x00051af4 chip=0x10021af4 rev=0x00

A step closer, I will keep trying.

1 Like
14

That's really all I have. I would recommend posting on the proxmox forums about this. The devs are relatively active on the forum and will be the best people to help you. Sorry I couldn't get you to a complete solution. :frowning:

15

I have given up on the idea that PFsense can use PCIe-passthrough in Proxmox, because after PFsense 2.1 the virtio modules are not included. It took a lot of time to figure it out but since we can't build PFsense with the modules we desire it's not possible. I will have to scrape some hardware together and install PFsense on the bare-metal.
Thanks to the people who helped me :wink: