Thunderbolt PCIe problems - MSIX PBA outside of specified BAR

Hi,

I’m up against a difficult problem here :slight_smile: Because I don’t want to change my existing setup, I found a theoretical setup and I’m almost gotten it to work.

As I don’t have any PCIe slots on my tiny home server, I figured I could use USB4/Thunderbolt3 GPU Dock and install a RAID card on it. As GPU’s work, so why not other cards, right? So far it seems that the problem is not with Software/OS/Kernel/Virtualization. As two separate Linux OS get the same error.

The GPU dock is TH3P4G3 with JHL7540 Intel Chip. Looks like some PCI cards work, but the RAID does not work (it works in a proper PCI slot)

So the error I get is hardware reports invalid configuration, MSIX PBA outside of specified BAR
dmesg | grep -iC 3 “07:00.0” gives me errors like

[    0.419100] system 00:00: [mem 0xe0000000-0xefffffff] has been reserved
[    0.421312] pnp 00:02: disabling [io  0x0010-0x001f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421314] pnp 00:02: disabling [io  0x0022-0x003f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421315] pnp 00:02: disabling [io  0x0063] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421316] pnp 00:02: disabling [io  0x0065] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421317] pnp 00:02: disabling [io  0x0067-0x006f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421318] pnp 00:02: disabling [io  0x0072-0x007f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421319] pnp 00:02: disabling [io  0x0080] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421319] pnp 00:02: disabling [io  0x0084-0x0086] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421320] pnp 00:02: disabling [io  0x0088] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421321] pnp 00:02: disabling [io  0x008c-0x008e] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421322] pnp 00:02: disabling [io  0x0090-0x009f] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421323] pnp 00:02: disabling [io  0x00a2-0x00bf] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421323] pnp 00:02: disabling [io  0x00b1] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421324] pnp 00:02: disabling [io  0x00e0-0x00ef] because it overlaps 0000:07:00.0 BAR 0 [io  0x0000-0x00ff]
[    0.421345] system 00:02: [io  0x04d0-0x04d1] has been reserved
[    0.421346] system 00:02: [io  0x040b] has been reserved
[    0.421347] system 00:02: [io  0x04d6] has been reserved
--
[    0.429821] pci 0000:06:01.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.429821] pci 0000:06:04.0: BAR 14: no space for [mem size 0x00200000]
[    0.429822] pci 0000:06:04.0: BAR 14: failed to assign [mem size 0x00200000]
[    0.429823] pci 0000:07:00.0: BAR 6: no space for [mem size 0x00100000 pref]
[    0.429824] pci 0000:07:00.0: BAR 6: failed to assign [mem size 0x00100000 pref]
[    0.429825] pci 0000:07:00.0: BAR 3: no space for [mem size 0x00040000 64bit]
[    0.429825] pci 0000:07:00.0: BAR 3: failed to assign [mem size 0x00040000 64bit]
[    0.429826] pci 0000:07:00.0: BAR 1: no space for [mem size 0x00010000 64bit]
[    0.429827] pci 0000:07:00.0: BAR 1: failed to assign [mem size 0x00010000 64bit]
[    0.429828] pci 0000:07:00.0: BAR 0: assigned [io  0xb000-0xb0ff]
[    0.429833] pci 0000:06:01.0: PCI bridge to [bus 07]
[    0.429836] pci 0000:06:01.0:   bridge window [io  0xb000-0xbfff]
[    0.429845] pci 0000:06:01.0:   bridge window [mem 0x7fb0000000-0x7fb01fffff 64bit pref]
--
[    0.430944] pci 0000:06:01.0: Adding to iommu group 6
[    0.430965] pci 0000:06:02.0: Adding to iommu group 6
[    0.430987] pci 0000:06:04.0: Adding to iommu group 6
[    0.431026] pci 0000:07:00.0: Adding to iommu group 6
[    0.431064] pci 0000:08:00.0: Adding to iommu group 6
[    0.431089] pci 0000:65:00.0: Adding to iommu group 18
[    0.431098] pci 0000:65:00.1: Adding to iommu group 19
--
[   87.003422] fwbr103i0: port 2(tap103i0) entered disabled state
[   87.003476] fwbr103i0: port 2(tap103i0) entered blocking state
[   87.003477] fwbr103i0: port 2(tap103i0) entered forwarding state
[   87.571868] vfio-pci 0000:07:00.0: enabling device (0000 -> 0001)
[   87.679810] vfio-pci 0000:07:00.0: vfio_ecap_init: hiding ecap 0x19@0x1e0
[   87.814619] fwbr103i0: port 2(tap103i0) entered disabled state
[   87.849916] fwbr103i0: port 1(fwln103i0) entered disabled state
[   87.849947] vmbr0: port 5(fwpr103p0) entered disabled state
--
[  206.357054] pcieport 0000:00:01.3:    [ 7] BadDLLP              
[  499.819389] r8169 0000:01:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
[  499.835760] r8169 0000:02:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
[  499.838390] vfio-pci 0000:07:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM

lspci -nn gives

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b5] (rev 01)
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b6]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:01.4 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:02.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:02.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14ba]
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14cd]
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:04.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14cd]
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b7] (rev 01)
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b9] (rev 10)
00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b9] (rev 10)
00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:14b9] (rev 10)
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 71)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1679]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167a]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167b]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167c]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167d]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167e]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:167f]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Device [1022:1680]
01:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
02:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125] (rev 05)
03:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/980PRO [144d:a80a]
04:00.0 Network controller [0280]: MEDIATEK Corp. Device [14c3:0608]
05:00.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
06:01.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
06:02.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
06:04.0 PCI bridge [0604]: Intel Corporation JHL7540 Thunderbolt 3 Bridge [Titan Ridge DD 2018] [8086:15ef] (rev 06)
07:00.0 Serial Attached SCSI controller [0107]: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0087] (rev 05)
08:00.0 USB controller [0c03]: Intel Corporation JHL7540 Thunderbolt 3 USB Controller [Titan Ridge DD 2018] [8086:15f0] (rev 06)
65:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1681] (rev d4)
65:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:1640]
65:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] VanGogh PSP/CCP [1022:1649]
65:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:161d]
65:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:161e]
65:00.5 Multimedia controller [0480]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/FireFlight/Renoir Audio Processor [1022:15e2] (rev 60)
65:00.6 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) HD Audio Controller [1022:15e3]
65:00.7 Signal processing controller [1180]: Advanced Micro Devices, Inc. [AMD] Raven/Raven2/Renoir Sensor Fusion Hub [1022:15e4]
66:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev a1)
67:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:161f]
67:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d6]
67:00.4 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:15d7]
67:00.5 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:162e]
67:00.6 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:162f]

The BIOS is some unrefined s***. There are some strange BUS size settings, but I have no glue what to change. The dock does not always work either, it stays in standby mode and with some bios setting combination it only turns on.

Maybe someone is some PCIe/Thunderbolt engineer who can say how to fix this problem? :grin:


There are also some USB4 pre-CM configurations, that I have to set to tunneling…

Oh, and the Raid nor Thunderbolt dock does not show up if I boot up a Windows…

So I finally found a simple solution. I had to go to the BIOS and change the PCIe BUS size.
These settings worked:

2 Likes