After testing the P411W-32P in several motherboards with the latest firmware (v4.1.3.1 - Shows up as v1.3.1) and the previous version (v4.1.2.1 - shows up as v0.1.2.0), I can definitively state the following…
If the motherboard firmware cannot see attached SSD, no OS will. Drivers won’t matter either.
The latest firmware hates Samsung SSDs. Nothing shows up regardless of OS or motherboard firmware. It plays a little bit nicer with WD SSDs and will at least show the (WD SandDisk) NVMe controller when attached, but still no storage medium.
I’m staring to think the particular reason why it works perfectly in the lab is because there aren’t any SSDs other than Seagates attached.
All in all, it is a firmware issue of some sort. Whether it is on the controllers’ end or the SSD, I won’t know until I have a larger sample size.
What I do know is:
Seagate has been using Phison for flash media controllers.
Both WD and Samsung are vertically integrated – they design everything top to bottom from flash media to controllers to firmware.
Also, no need to wipe everything in order to downgrade firmware on the P411W-32P, at least in my most recent testing.
Thanks for that additional data! Just so that there is no misunderstanding on my end you have already been able to test the P411W-32P on all the motherboards you listed?
In the meantime I was able to confirm my findings with another AM4 motherboard, an ASUS ProArt X570-CREATOR WIFI with UEFI 0801 and a 3900 Pro. Was the first time I was able to test X570+PCIe Gen4+AGESA 1207, had hoped that maybe an AGESA update would help out here.
Yes, I see the same behavior with simple firmware downgrading (=restoring functionality) not needing additional erase steps.
Interestingly I cannot confirm WD SSDs showing up at all with firmware 4.1.3.1 (0.1.3.1), tested:
SN550 2 TB
SN750 2 TB
Ultrastar DC SN630 7.68 TB
They only ever show up with P411W-32P firmware up to version 4.1.2.1, exactly the same as with the Samsung PM1733 7.68 TB units. Have you been able to use any SSD with firmware 4.1.3.1?
BUT I saw a crashed state of the P411W-32P after extensive S3 (Sleep/Suspend-to-RAM) testing where after reawakening the system with active SSD file transfers which had been paused by the system going sleep the SSD showed up as a PCIe device but wasn’t accessable as storage any more.
Rebooting the system fixed that again.
The same SSDs attached to the same motherboards directly with passive adapters function absolutely stable with S3 sleep.
This crashed state seems to occur more often with Broadcom’s drivers installed instead of letting the P411W-32P remain as an unkown “Base System Device”.
For me this is the main reason for pursuing Broadcom to fix the firmware and/or drivers since I want to use the P411W-32P in a workstation scenario where the system is also allowed to go to sleep.
My curiosity* got the better of me and I ordered a Broadcom HBA 9500 16i. If you have any tests I could help you out with while you have to wait for yours just drop a message here.
*What I want to find out:
Does it have the same firmware issues as the ones from the 9400 8i8e I’ve been experiencing?
Performance comparison Broadcom HBA 9500 16i vs. P411W-32P when using NVMe SSDs?
Neat thing: With active HBAs you can use PCIe Gen4 SSDs without automatically cutting their performance on Gen3 systems like current Ryzen APU builds.
Does the 9500 16i (as a “Tri-Mode” controller) really not work with Icy Dock’s SFF-8643 and OCuLink U.2 NVMe backplanes?
If the 9500 16i turns out to also be shitty I think I might get the closure I need to finally move on from Broadcom for good…
Hopefully the card will arrive in 1-2 weeks. Unfortunately the P5800X order that said delivery date start of June when I ordered it, now has a new delivery date of Jan 2023
I’m awaiting some explanation.
I wonder if I should order a Micron 7450 U.3, though I’ve only seen the 7400 on the supported NVMe drive list. All my other SSDs are M.2 installed in the Asus Hyper M.2 x16 Gen4 card (or a very old SATA SSD in my old PC).
I think the next SSD for “serious” data handling I’ll be looking at will be from Micron, too.
(-) Not the fastest
(+) Good value
(+) Anecdotal personal evidence: Never had issues with Micron SSDs, had a WD SN630 “softfail”, meaning it lost its data (even SMART) and it wasn’t accessable until it got secure-erased.
(+) Firmware updates available for retail end customers (not with Samsung’s DC SSDs)
(+) 5 years warranty available for retail end customers (not with Samsung’s DC SSDs)
Ever since the P4500/P4510 debacle I don’t trust Intel NAND SSDs anymore (see also my previous Broadcom NVMe HBA experiences), Samsung are dicks regarding OEM SSDs and end customer service, Kioxa is a bit higher priced (in Germany), WD is a bit “retardé” so Micron seems the only one left.
My 9500-16i arrived today, had a quick look at it:
The issues I’ve been having with my two 9400-8i8es are also occurring in the exact same way with the 9500-16i.
Wouldn’t use a Tri-Mode HBA for NVMe SSDs, the two generations I could test so far (Broadcom 9400 and 9500) eat up quite a lot of performance compared to the pure PCIe Switches.
Confirmed: Icy Dock’s OCuLink backplane doesn’t work with the 9500-16i.
Will report back wíth more details after I’ve had more time with the 9500.
Those were the issues with the P4500/P4510 SSDs? Have you ever heard back from Broadcom/Intel after the other thread was closed. I’m a little puzzled by how few drives are actually supported officially according to broadcom’s official supported drive list.
ok. I guess this is because the PCIe Switches are using x16 instead of x8 PCIe bus width + some overhead?
Ok, I’m currently not planning to use icydock enclosures. What cable did you use to connect?
My card is supposed to arrive tomorrow, but my SSD I’ve ordered won’t be delivered until Jan 2023 according to Digitec. Gee thanks digitec for wrong delivery dates on homepage!
Nope, the P4500 et al. issues were entirely faulty SSD model lines from Intel with bugs that when triggered resulted in extremely reduced performance.
My current woes with the 9400-8i8e or the 9500-16i are when using Windows in combination with system sleep (S3), after waking the system up the HBAs are in a crashed state and a BSOD follows as soon as you try using a connected drive. Like the similar P411W-32P issues the frequency of the BSODs depends on the used Broadcom driver version.
Broadcom puts its head in sand here, support person cannot find anything wrong.
My current opinion is that the PCIe Switch HBAs are faster since they are just that, PCIe switches. NVMe SSDs show up the exact same way as they would when installed directly on a motherboard.
The Tri-Mode controllers on the other hand add a layer of abstraction to the drives, there the drives are actually managed by Broadcom’s drivers and firmware. SMART Monitoring tools like CrystalDiskInfo cannot detect SATA or NVMe drives connected to Tri-Mode HBAs which kind of sucks.
The very same Broadcom 05-60001-00 I’ve been using successfully with the P411W-32P.
Will post concrete numbers hopefully soon, have to wait until I have a spare Zen 3 CPU to give the HBAs the “best” possible performance scenarios. Some SSD benchmarks are actually CPU-limited.
I think (Just a gut feeling) the source of the issues with the Icy Dock OCuLink backplane is that Tri-Mode HBAs cannot detect/sense the backplane to have PCIe/NVMe support. This is why it works fine when using it with a P411W-32P that only supports PCIe.
Tri-Mode 9400/9500 HBAs also have something weird going on with their firmware packages: You have to choose between a “Mixed support” (= it can handle SATA/SAS and NVMe drives) and a package that only supports SATA and SAS.
Doesn’t seem to be a thing from local shops here and the only way I can test the 9500-16i with SATA drives at the moment is indirectly with a SAS3 expander that unfortunately tops a single drive’s performance at around 510 MB/s.
Seems I have to get a 2xSFF-8643 backplane to test SATA SSDs without logic betweeen the HBA and the drives themselves.
The tests I’m doing here are just a simple bare metal Windows 10 21H2 installation.
If you have a system that’s running Windows natively and supports S3 (“real” Server motherboards have S3 disabled on a BIOS level) could you please do a quick test if you also get a BSOD (“DRIVER POWER STATE FAILURE”) when using a drive on the 9500-16i (with a current Broadcom driver, not the generic Windows driver from 2017) after the system was in S3 sleep?
I tested the P411W-32P on the following
Main System, Spare-1, Spare-2, Spare-3, and Spare-4
AND
1x Samsung DCT 983, 1x WD Blue SN550, 1x Samsung 970 Evo Plus
I eventually gave up after seeing the same results over and over again. The issue looks to be motherboard and OS independent.
The OS used is Linux Mint 20.3 with kernel 5.13.xx something.
Since none of the SSD’s showed up in the GUI, I resorted to command line - lspci
That’s when I noticed the WD/Sandisk SSD controller at least making an appearance.
I wish it would. I really do. It’s seriously looking like a firmware issue on either the SSDs we pick or the P411W-32P. I’m banking on the latter.
Absolutely not a single one. I might have a MSI Spatium M470 1TB based on a Phison E16 controller hanging around I might be able to test… Time to see what Windows 10 21H2 thinks of the P411W-32P fw v4.1.3.1 (0.1.3.1) and a WD SSD (and perhaps that Spatium too) on Spare-1.
That sounds rather familiar to what was happening with the WD SSD, but without the S3 states.
Thanks again for your extensive testing. May I use your testing results to build up my support case with Broadcom? If I get any (beta) firmware update I’ll relay it here.
I just thought of a test to see if the firmware has legitimately crashed.
If I recall correctly, the P411W-32P supports hotswap. So, to test, perform a hotswap/hotadd to see if it really has crashed…
When I’m having the P411W-32P issues an obvious indicator for the crashed state is that the g4xflash tool can’t talk to the P411W-32P anymore until the system is rebooted.
Unfortunately I don’t have any Windows running natively, only virtualized.
So far it looks good with the new controller.
CLI Version = 007.2106.0000.0000 Feb 28, 2022
Operating system = Linux 5.16.0-0.bpo.4-amd64
Controller = 0
Status = Success
Description = None
Product Name = HBA 9500-16i
PCI Address = 00:61:00:00
System Time = 06/16/2022 02:39:41
FW Package Build = 14.00.00.00
FW Version = 14.00.00.00
BIOS Version = 09.27.00.00_14.00.00.00
NVDATA Version = 14.11.00.07
Driver Name = mpt3sas
Driver Version = 39.100.00.00
PD LIST :
EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp
The only thing is that my second old LSI HBA controller doesn’t get listed anymore with lspci.
But I guess something came lose during installation. So I will have to check (it’s blocked by 2nd vertical mounted GPU for passthrough. So I have disassemble stuff again )
Effective! I had also forgotten about the firmware status LED too. lol
I just completed some more testing with the P411W-32P on Spare-1 and the MSI Spatium M470 SSD mentioned earlier. Everything was bare metal – No virtualization here. (Begs the question though… How well will this card plays with PCIe passthrough & SR-IOV? )
Still no bones getting anything to show up with fw v4.1.3.1.
Well… Partially as mentioned before. This time I made sure to take better records and screenshots.
Here is what Linux spat out with the bugged firmware (v4.1.3.1) with a WD SN550 and a Samsung DCT 983 attached to the card:
*Important entries have been bolded
**Please note that the italicized entries are directly attached to the motherboard and contain the OS.
Windows Enterprise LTSC 21H2 was used to further show what was going on.
The WD SN550 controller shows up, but no storage device. No drivers were installed in this screen cap. Also, a little bit of good news… PCI AER looks to be enabled from the card to the SSD even if the motherboard does not enable nor support it. If anyone wants to try and confirm functionality of this, by all means be please do.
