A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

This post will serve as my hardware catalog.

Components Main System Server/NAS Firewall Spare-1 Spare-2 Spare-3 Spare-4 Spare-5
CPU AMD Ryzen 9 5900X Intel Xeon E-2136 Intel i3-6100 Intel Xeon E3-1220 V5 AMD Ryzen 7 1700 AMD Ryzen 3 1200 Intel i7-6800K Intel i5-2400
Motherboard Asus ROG CrossHair VIII Dark Hero SuperMicro X11SCL-F Dell OptiPlex 3040 SuperMicro X11SSL-F Asus CrossHair VI Hero ASRock AB350 Pro4 ATX Asus Sabertooth x99 Dell OptiPlex 790
Motherboard Firmware 4201 1.6 1.12.3 2.6 7901 P5.8 4101
Chipset AMD x570 Intel C242 Intel C232 AMD x370 AMD B350 Intel X99
Memory 2x 32GB Samsung M391A4G43AB1-CWE ECC UDIMM 2x 32GB Samsung M391A4G43AB1-CVF ECC UDIMM 8GB
GPU NVIDIA GeForce RTX 3080 in PCIe 4 x8 ASPEED AST2500 BMC CPU Integrated ASPEED AST2400 BMC CPU Integrated
--------------------------------------------------------------------------------------------
SSDs Capacity Quantity
Samsung 980 Pro 2000GB 2
Samsung 980 DCT 960GB 5
WD Blue SN550 1000GB 1
WD Black SN850 1000GB 1
1 Like

Finally, results of testing!

After testing the P411W-32P in several motherboards with the latest firmware (v4.1.3.1 - Shows up as v1.3.1) and the previous version (v4.1.2.1 - shows up as v0.1.2.0), I can definitively state the following…

If the motherboard firmware cannot see attached SSD, no OS will. Drivers won’t matter either.
The latest firmware hates Samsung SSDs. Nothing shows up regardless of OS or motherboard firmware. It plays a little bit nicer with WD SSDs and will at least show the (WD SandDisk) NVMe controller when attached, but still no storage medium.

I’m staring to think the particular reason why it works perfectly in the lab is because there aren’t any SSDs other than Seagates attached.

All in all, it is a firmware issue of some sort. Whether it is on the controllers’ end or the SSD, I won’t know until I have a larger sample size.
What I do know is:

  1. Seagate has been using Phison for flash media controllers.
  2. Both WD and Samsung are vertically integrated – they design everything top to bottom from flash media to controllers to firmware.

Also, no need to wipe everything in order to downgrade firmware on the P411W-32P, at least in my most recent testing.

3 Likes

@Illumous

Thanks for that additional data! Just so that there is no misunderstanding on my end you have already been able to test the P411W-32P on all the motherboards you listed?

In the meantime I was able to confirm my findings with another AM4 motherboard, an ASUS ProArt X570-CREATOR WIFI with UEFI 0801 and a 3900 Pro. Was the first time I was able to test X570+PCIe Gen4+AGESA 1207, had hoped that maybe an AGESA update would help out here.

Yes, I see the same behavior with simple firmware downgrading (=restoring functionality) not needing additional erase steps.

Interestingly I cannot confirm WD SSDs showing up at all with firmware 4.1.3.1 (0.1.3.1), tested:

  • SN550 2 TB
  • SN750 2 TB
  • Ultrastar DC SN630 7.68 TB

They only ever show up with P411W-32P firmware up to version 4.1.2.1, exactly the same as with the Samsung PM1733 7.68 TB units. Have you been able to use any SSD with firmware 4.1.3.1?

BUT I saw a crashed state of the P411W-32P after extensive S3 (Sleep/Suspend-to-RAM) testing where after reawakening the system with active SSD file transfers which had been paused by the system going sleep the SSD showed up as a PCIe device but wasn’t accessable as storage any more.

Rebooting the system fixed that again.

The same SSDs attached to the same motherboards directly with passive adapters function absolutely stable with S3 sleep.

This crashed state seems to occur more often with Broadcom’s drivers installed instead of letting the P411W-32P remain as an unkown “Base System Device”.

For me this is the main reason for pursuing Broadcom to fix the firmware and/or drivers since I want to use the P411W-32P in a workstation scenario where the system is also allowed to go to sleep.

1 Like

My curiosity* got the better of me and I ordered a Broadcom HBA 9500 16i. If you have any tests I could help you out with while you have to wait for yours just drop a message here.

*What I want to find out:

  1. Does it have the same firmware issues as the ones from the 9400 8i8e I’ve been experiencing?

  2. Performance comparison Broadcom HBA 9500 16i vs. P411W-32P when using NVMe SSDs?

Neat thing: With active HBAs you can use PCIe Gen4 SSDs without automatically cutting their performance on Gen3 systems like current Ryzen APU builds.

  1. Does the 9500 16i (as a “Tri-Mode” controller) really not work with Icy Dock’s SFF-8643 and OCuLink U.2 NVMe backplanes?

If the 9500 16i turns out to also be shitty I think I might get the closure I need to finally move on from Broadcom for good…

Thanks for the offer to help.

Hopefully the card will arrive in 1-2 weeks. Unfortunately the P5800X order that said delivery date start of June when I ordered it, now has a new delivery date of Jan 2023 :rage:
I’m awaiting some explanation.

I wonder if I should order a Micron 7450 U.3, though I’ve only seen the 7400 on the supported NVMe drive list. All my other SSDs are M.2 installed in the Asus Hyper M.2 x16 Gen4 card (or a very old SATA SSD in my old PC).

1 Like

I think the next SSD for “serious” data handling I’ll be looking at will be from Micron, too.

(-) Not the fastest

(+) Good value
(+) Anecdotal personal evidence: Never had issues with Micron SSDs, had a WD SN630 “softfail”, meaning it lost its data (even SMART) and it wasn’t accessable until it got secure-erased.
(+) Firmware updates available for retail end customers (not with Samsung’s DC SSDs)
(+) 5 years warranty available for retail end customers (not with Samsung’s DC SSDs)

Ever since the P4500/P4510 debacle I don’t trust Intel NAND SSDs anymore (see also my previous Broadcom NVMe HBA experiences), Samsung are dicks regarding OEM SSDs and end customer service, Kioxa is a bit higher priced (in Germany), WD is a bit “retardé” so Micron seems the only one left.

1 Like

My 9500-16i arrived today, had a quick look at it:

  • The issues I’ve been having with my two 9400-8i8es are also occurring in the exact same way with the 9500-16i.

  • Wouldn’t use a Tri-Mode HBA for NVMe SSDs, the two generations I could test so far (Broadcom 9400 and 9500) eat up quite a lot of performance compared to the pure PCIe Switches.

  • Confirmed: Icy Dock’s OCuLink backplane doesn’t work with the 9500-16i. :frowning:

Will report back wíth more details after I’ve had more time with the 9500.

2 Likes

Thank you very much

Those were the issues with the P4500/P4510 SSDs? Have you ever heard back from Broadcom/Intel after the other thread was closed. I’m a little puzzled by how few drives are actually supported officially according to broadcom’s official supported drive list.

ok. I guess this is because the PCIe Switches are using x16 instead of x8 PCIe bus width + some overhead?

Ok, I’m currently not planning to use icydock enclosures. What cable did you use to connect?

My card is supposed to arrive tomorrow, but my SSD I’ve ordered won’t be delivered until Jan 2023 according to Digitec. Gee thanks digitec for wrong delivery dates on homepage! :roll_eyes:

Nope, the P4500 et al. issues were entirely faulty SSD model lines from Intel with bugs that when triggered resulted in extremely reduced performance.

My current woes with the 9400-8i8e or the 9500-16i are when using Windows in combination with system sleep (S3), after waking the system up the HBAs are in a crashed state and a BSOD follows as soon as you try using a connected drive. Like the similar P411W-32P issues the frequency of the BSODs depends on the used Broadcom driver version.

Broadcom puts its head in sand here, support person cannot find anything wrong.

My current opinion is that the PCIe Switch HBAs are faster since they are just that, PCIe switches. NVMe SSDs show up the exact same way as they would when installed directly on a motherboard.

The Tri-Mode controllers on the other hand add a layer of abstraction to the drives, there the drives are actually managed by Broadcom’s drivers and firmware. SMART Monitoring tools like CrystalDiskInfo cannot detect SATA or NVMe drives connected to Tri-Mode HBAs which kind of sucks.

The very same Broadcom 05-60001-00 I’ve been using successfully with the P411W-32P.

Will post concrete numbers hopefully soon, have to wait until I have a spare Zen 3 CPU to give the HBAs the “best” possible performance scenarios. Some SSD benchmarks are actually CPU-limited.

I think (Just a gut feeling) the source of the issues with the Icy Dock OCuLink backplane is that Tri-Mode HBAs cannot detect/sense the backplane to have PCIe/NVMe support. This is why it works fine when using it with a P411W-32P that only supports PCIe.

Tri-Mode 9400/9500 HBAs also have something weird going on with their firmware packages: You have to choose between a “Mixed support” (= it can handle SATA/SAS and NVMe drives) and a package that only supports SATA and SAS.

I have no idea what that’s about.

1 Like

Has anybody seen an SFF-8654 8i-to-8xSATA cable?

Doesn’t seem to be a thing from local shops here and the only way I can test the 9500-16i with SATA drives at the moment is indirectly with a SAS3 expander that unfortunately tops a single drive’s performance at around 510 MB/s.

Seems I have to get a 2xSFF-8643 backplane to test SATA SSDs without logic betweeen the HBA and the drives themselves.

There’s the 05-60006-00 x8 SFF-8654 → Eight U.3 SFF-8639 Cable.

My card finally arrived today, so I can do a test in the next 1-2 days.
To my surprise the P5800X also arrived.

So I can test
9500-16i → 05-60006-00 (U.3) → SATA HDDs
9500-16i → 05-60005-00 (U.2) → P5800X SSD

My box never sleeps :wink:
Windows is the main operating system, or do you run it virtualized?

1 Like

Happy to hear your toys arrived!

The tests I’m doing here are just a simple bare metal Windows 10 21H2 installation.

If you have a system that’s running Windows natively and supports S3 (“real” Server motherboards have S3 disabled on a BIOS level) could you please do a quick test if you also get a BSOD (“DRIVER POWER STATE FAILURE”) when using a drive on the 9500-16i (with a current Broadcom driver, not the generic Windows driver from 2017) after the system was in S3 sleep?

I tested the P411W-32P on the following
Main System, Spare-1, Spare-2, Spare-3, and Spare-4
AND
1x Samsung DCT 983, 1x WD Blue SN550, 1x Samsung 970 Evo Plus

I eventually gave up after seeing the same results over and over again. The issue looks to be motherboard and OS independent.
The OS used is Linux Mint 20.3 with kernel 5.13.xx something.
Since none of the SSD’s showed up in the GUI, I resorted to command line - lspci
That’s when I noticed the WD/Sandisk SSD controller at least making an appearance.

I wish it would. I really do. It’s seriously looking like a firmware issue on either the SSDs we pick or the P411W-32P. I’m banking on the latter.

Absolutely not a single one. I might have a MSI Spatium M470 1TB based on a Phison E16 controller hanging around I might be able to test… Time to see what Windows 10 21H2 thinks of the P411W-32P fw v4.1.3.1 (0.1.3.1) and a WD SSD (and perhaps that Spatium too) on Spare-1.

That sounds rather familiar to what was happening with the WD SSD, but without the S3 states.

1 Like

@Illumous

Thanks again for your extensive testing. May I use your testing results to build up my support case with Broadcom? If I get any (beta) firmware update I’ll relay it here.

1 Like

You’re welcome.
And
Knock yourself out XD

1 Like

You could also open a support ticket with Broadcom, maybe you’ll get a more motivated support human by chance :stuck_out_tongue:

1 Like

@aBav.Normie-Pleb
This is true… To the lottery anyone? lol

I just thought of a test to see if the firmware has legitimately crashed.
If I recall correctly, the P411W-32P supports hotswap. So, to test, perform a hotswap/hotadd to see if it really has crashed… :thinking:

When I’m having the P411W-32P issues an obvious indicator for the crashed state is that the g4xflash tool can’t talk to the P411W-32P anymore until the system is rebooted.

1 Like

Unfortunately I don’t have any Windows running natively, only virtualized.

So far it looks good with the new controller.

CLI Version = 007.2106.0000.0000 Feb 28, 2022
Operating system = Linux 5.16.0-0.bpo.4-amd64
Controller = 0
Status = Success
Description = None

Product Name = HBA 9500-16i
PCI Address = 00:61:00:00
System Time = 06/16/2022 02:39:41
FW Package Build = 14.00.00.00
FW Version = 14.00.00.00
BIOS Version = 09.27.00.00_14.00.00.00
NVDATA Version = 14.11.00.07
Driver Name = mpt3sas
Driver Version = 39.100.00.00

PD LIST :


EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp

0:6 3 JBOD - 372.611 GB NVMe SSD - - 512B INTEL SSDPF21Q400GB -
0:8 1 JBOD - 16.370 TB SATA HDD - - 4 KB ST18000NM000J-2TV103 -
0:9 2 JBOD - 16.370 TB SATA HDD - - 4 KB ST18000NM000J-2TV103 -

The only thing is that my second old LSI HBA controller doesn’t get listed anymore with lspci.
But I guess something came lose during installation. So I will have to check (it’s blocked by 2nd vertical mounted GPU for passthrough. So I have disassemble stuff again :frowning: )

1 Like

Effective! I had also forgotten about the firmware status LED too. lol :person_facepalming:

I just completed some more testing with the P411W-32P on Spare-1 and the MSI Spatium M470 SSD mentioned earlier. Everything was bare metal – No virtualization here. (Begs the question though… How well will this card plays with PCIe passthrough & SR-IOV? :thinking:)
Still no bones getting anything to show up with fw v4.1.3.1.

Well… Partially as mentioned before. This time I made sure to take better records and screenshots.

Here is what Linux spat out with the bugged firmware (v4.1.3.1) with a WD SN550 and a Samsung DCT 983 attached to the card:

[email protected]:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07)
00:13.0 Non-VGA unclassified device: Intel Corporation 100 Series/C230 Series Chipset Family Integrated Sensor Hub (rev 31)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)
00:1d.1 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #10 (rev f1)
00:1d.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #11 (rev f1)
00:1f.0 ISA bridge: Intel Corporation C232 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:00.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:01.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:02.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:03.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:04.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:05.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:06.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:07.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:1f.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
03:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
04:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
05:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
06:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
07:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
08:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
09:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
0a:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
0b:00.0 Serial Attached SCSI controller: Broadcom / LSI Device 00b2 (rev b0)
0c:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
0d:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0e:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0f:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
10:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

Here is what Linux spat out with firmware v4.1.2.1 with a WD SN550 and a Samsung DCT 983 attached to the card:

[email protected]:~# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07)
00:13.0 Non-VGA unclassified device: Intel Corporation 100 Series/C230 Series Chipset Family Integrated Sensor Hub (rev 31)
00:14.0 USB controller: Intel Corporation 100 Series/C230 Series Chipset Family USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation 100 Series/C230 Series Chipset Family Thermal Subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation 100 Series/C230 Series Chipset Family MEI Controller #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode] (rev 31)
00:1d.0 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #9 (rev f1)
00:1d.1 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #10 (rev f1)
00:1d.2 PCI bridge: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #11 (rev f1)
00:1f.0 ISA bridge: Intel Corporation C232 Chipset LPC/eSPI Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation 100 Series/C230 Series Chipset Family Power Management Controller (rev 31)
00:1f.4 SMBus: Intel Corporation 100 Series/C230 Series Chipset Family SMBus (rev 31)
01:00.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:00.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:01.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:02.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:03.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:04.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:05.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:06.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:07.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
02:1f.0 PCI bridge: Broadcom / LSI Device c012 (rev b0)
03:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
04:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
05:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
06:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
07:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
08:00.0 System peripheral: Broadcom / LSI Device 02b2 (rev b0)
09:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
0a:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
0b:00.0 Serial Attached SCSI controller: Broadcom / LSI Device 00b2 (rev b0)
0c:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
0d:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0e:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
0f:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 03)
10:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 30)

*Important entries have been bolded
**Please note that the italicized entries are directly attached to the motherboard and contain the OS.

Windows Enterprise LTSC 21H2 was used to further show what was going on.
The WD SN550 controller shows up, but no storage device. No drivers were installed in this screen cap. Also, a little bit of good news… PCI AER looks to be enabled from the card to the SSD even if the motherboard does not enable nor support it. If anyone wants to try and confirm functionality of this, by all means be please do.

Previous drivers work with the latest fw. I should have shown the deets there. Oops :person_shrugging:

Here’s what it should look like with everything working proper. fw v4.1.2.1 and previous drivers

On a different note,
For those interested, here is a link to the cables I’ve purchased. PCIe 4 rated and only about $35 USD at the time of purchase and typing.
https://www.bhphotovideo.com/c/product/1629046-REG/highpoint_ts8i_8639_060_sff_8654_to_u_2_sff_8639.html

1 Like