Hi,
Setting up my first NAS, with proxmox, zfs… very similar to this other guides in this forum.
I have a x470 Taichi motherboard, an IBM M1015 SAS HBA which I bought pre flashed to IT mode firmware and i can’t get it to work at PCIe 8x. I know it’s supposed to work at that speed but no matter which slot I plug it in, it connects with 4x lanes.
The first two MB slots are directly connected to the CPU, they’re supposed to work at 16x + 4x or 8x + 8x. I started with a Chelsio PCIe3.0 8x NIC on the first slot, this PCIe2.0 8x LSI HBA on the second slot, and low end Geforce GPU (just for setting up the server) at the last, south-bridge connected slot (PCIe2.0 4x).
I’ve tried changing the AMD PBS / PCI x16/2x8 Switch BIOS setting to 2x8.
Now I’ve pulled the NIC card from the first slot and let the HBA get the first slot, and set the slot back to 1x16x.
I’ve gone through the trouble of flashing the firmware (from 20.00.00.00 to 20.00.07.00).
Tried all those things, but still, if I run dmseg | band I get:
[ 0.180750] pci 0000:03:00.0: 8.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x2 link at 0000:02:00.0 (capable of 15.752 Gb/s with 8 GT/s x2 link)
[ 0.184052] pci 0000:0b:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x4 link at 0000:02:04.0 (capable of 32.000 Gb/s with 5 GT/s x8 link)
[ 0.184939] pci 0000:0d:00.0: 16.000 Gb/s available PCIe bandwidth, limited by 5 GT/s x4 link at 0000:00:03.1 (capable of 32.000 Gb/s with 5 GT/s x8 link)
The last line refers to the HBA. The one before to the GPU, I’m not sure about the others.
lspci -s 0000:0d:00.0 -vvv (summarized)
0d:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
Subsystem: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns, L1 <1us
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkSta: Speed 5GT/s, Width x4, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
Kernel driver in use: mpt3sas
Kernel modules: mpt3sas
lspci -tv
-[0000:00]-+-00.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Root Complex
+-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) I/O Memory Management Unit
+-01.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+-01.3-[01-0c]--+-00.0 Advanced Micro Devices, Inc. [AMD] Device 43d0
| +-00.1 Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller
| \-00.2-[02-0c]--+-00.0-[03]----00.0 ASMedia Technology Inc. ASM2142 USB 3.1 Host Controller
| +-02.0-[04]----00.0 ASMedia Technology Inc. ASM1062 Serial ATA Controller
| +-03.0-[05-0a]----00.0-[06-0a]--+-01.0-[07]----00.0 Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak]
| | +-03.0-[08]--
| | +-05.0-[09]----00.0 Intel Corporation I211 Gigabit Network Connection
| | \-07.0-[0a]--
| +-04.0-[0b]--+-00.0 NVIDIA Corporation GK208B [GeForce GT 710]
| | \-00.1 NVIDIA Corporation GK208 HDMI/DP Audio Controller
| \-09.0-[0c]--
+-02.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+-03.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+-03.1-[0d]----00.0 Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
+-04.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+-07.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+-07.1-[0e]--+-00.0 Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function
| +-00.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor
| \-00.3 Advanced Micro Devices, Inc. [AMD] Zeppelin USB 3.0 Host controller
+-08.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-1fh) PCIe Dummy Host Bridge
+-08.1-[0f]--+-00.0 Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function
| \-00.2 Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
+-14.0 Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller
+-14.3 Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge
+-18.0 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0
+-18.1 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1
+-18.2 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2
+-18.3 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3
+-18.4 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4
+-18.5 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5
+-18.6 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6
\-18.7 Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7
So my question is… did I buy a bad card? Is there anything else I might have to change in the BIOS to enable 8 lanes for this card?
And… Does it really matter? I have zfs working, but I was about to start testing, benchmarking and optimizing and I wanted to start from a good base. I don’t know if 4x PCIe 2.0 lanes are going to be a bottleneck, or add latency, or make my tests wrong. It’s just 8 hard drives connected to it so it might not matter, but I know this card works at 8x for other people so it bugs me…
