A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

[aBav.Normie-Pleb]

Smallest update:

1. Broadcom received my unit in their RMA facility in the Netherlands on May 11th, 2022 with the option send the replacement product as soon as the initial one was received and checked for not having any obvious physical damage.

2. Today (May 19th, 2022), I received a notification that the replacement unit has been shipped… from the US - I hope that doesn’t take long and that I’m not supposed to pay any customs fees for that package

At least I can be sure that they are not sending the old one back…

[Illumous]

You’re welcome for the details.
Seagate SSD’s eh…?
The SSD’s I used for testing were Samsung DCT 983’s (960GB, u.2, pcie 3, NVME 1.2b), and Samsung 980 Pro’s (2TB, m.2, pcie 4, NVME 1.3c). It seems ill have to branch out and test the WD Blue sn550, WD Black sn750, and other SSD’s that I have laying around… Time to buy an Epyc and the Micron 7400 Pro’s I was eyeballing too… For Science!!! And good measure!! (This is a few months out right now.)

Something to note about the P411W-32P… Put a fan on it! It gets H O T!!!

I see a few ways to go forward if the latest and greatest firmware is desired…

1. Wait for a new release.
2. Try systems by other vendors.
3. Try various SSD’s.
4. Some combination of the above…

@aBav.Normie-Pleb

when I first bought the 9400 8i8e I wasn’t aware that you need special Broadcom cables

Non-standard pinouts… Hmmm… I’ll have to investigate when I grab a tri-mode card after the prices come down on ebay…

@aBav.Normie-Pleb

Could I ask you for assistance regarding HBA flashing?

Yes. You can ask, however, I wish I could be more helpful with the 9400 series. I currently lack one to experiment with and the tools for this generation have changed. From what I’ve heard, the tools for my SAS3008 cards most likely won’t work for the 9400 series.

My flashing adventures started on a different forum and thread. It is my hope that this helps. This one is one of the threads:
https://forums.servethehome.com/index.php?threads/the-versatile-sas3008-chipset-my-vendor-crossflashing-adventures.28297/
The author, Sleyk, has been through the gauntlet with flashing various cards by many vendors.

@aBav.Normie-Pleb

the firmwares to flash consist of three parts (“Firmware”, BIOS" and “EFI”, a bit annoying)

From what you’ve said and the contents of the firmware update archive, it looks rather familiar to the SAS3008 based cards I have around.
The “Firmware” portion is required for card operation. The “BIOS” and “EFI” portions are not. Those two are what enables the card to be booted from like you would a “normal” storage device and allow configuring different aspects of the cards functionality, if I recall correctly. Wiping those out if you don’t intend to boot from it ‘may’ improve POST times too.

[Illumous]

If I may suggest, when the new card arrives:

1. Do no update it yet.
2. Install card and plug in SSDs.
3. See if SSDs show up.
   3a. If everything shows up proper, go to step 4.
   3b. If nothing that’s attached shows up
   3c. Troubleshoot.
   3d. Try updating or downgrading firmware.
4. Try updating firmware to v4.1.2.1
5. If that still works, evaluate risk of updating to v4.1.3.1

[aBav.Normie-Pleb]

Replacement P411W-32P has arrived today, immediate differences:

• Good: Samsung PM1733 U.2 SSDs now show up in the UEFI (with a SFF8654 8i-to-2x-SFF8639 (U.2) cable, Broadcom part number 05-60005-00);

• Bad: Windows 10 Enterprise 21H2 crashes with a BSOD (“System Thread Exception Not Handled” (?), displayed only for a very short amount of time) if the P411W-32P is installed in any PCIe slot, this never happened before and happens on both my test systems:

@aBav.Normie-Pleb

Platform A: ASUS ProArt B550 CREATOR/5750G/128 GB ECC Micron; HBA in first x16 PCIe slot (running at PCIe 3.0 x16);
Platform B: ASRock X570 Taichi/3700X/64 GB ECC Samsung; HBA in first x16 PCIe slot (running at PCIe 4.0 x8);

Yes, I am aware the P411W-32P runs quite hot, its typical power draw is rated at 21 W, about twice as high as Broadcom’s current PCIe x8 Tri-Mode HBAs; it has its own Noctua NF-F12 and the reported PCIe Switch Chipset temperature during operation was around 50°C (with my first unit).

Will check in again after I’ve tried re-installing Windows, hope it’s that simple BUT I’m a bit worried since on both my test systems Windows is installed on a SATA SSD running in AHCI mode, so something hardware-wise might be up if the P411W-32P causes booting BSODs even though it doesn’t handle the boot drive

[aBav.Normie-Pleb]

Further Broadcom P411W-32P development:

The cause for the BSOD during Windows’ boot is Broadcom’s drivers for the P411W-32P, could take a photo of one:

So logically the issue appeared on both my test platforms where I had previously tested the first P411W-32P.

Small victory: Just as with the Delock 90504 PCIe Gen3 Switch Adapter, the P411W-32P works without installing Windows drivers, but Windows Device Manager looks ugly with the multiple unkown “Base System Devices”

Don’t have time for additional testing today, but am cautiously optimistic…
As always, will give an situational update with any new development.

@Illumous

Does your P411W-32P work with an older Firmware version installed but Windows using the latest drivers?

[aBav.Normie-Pleb]

Thanks for your time explaining things!

I’m starting to “learn” the StorCLI64 stuff to be at least able to do manual firmware updates on my HBA 9400s since Broadcom’s LSA web interface software is also a bug fest (small highlight, only works OOB on English Windows installations, otherwise you have to edit a .conf file).

Yay, I can do the firmware updates for all (?) three parts of the firmware, but I also read up on erasing firmware to potentially fix the issues I’ve been experiencing.

In The guide for the Tri-Mode Adapters ( https://docs.broadcom.com/doc/MR-TM-StorCLI-UG - only found one for the HW RAID variant but firmware stuff should be similar) when looking at erasing parts there is a mentioning of erase exceptions for a “manufacturing data region”, do you have an idea what this is about? So there are 4 different firmware parts?

1. “Firmware” for general operation;
2. “BIOS” for boot support on non-UEFI systems;
3. “EFI” for boot support on UEFI systems;
4. “Manufacturing Data Region” for… ?

[aBav.Normie-Pleb]

CRITICAL Broadcom P411W-32P update:

I just looove Broadcom (and dislike it very, very much that they are to buy VMware).

Now, that I have a second unit back from RMA I could confirm @Illumous findings regarding the generally broken firmware with the latest version and additionally I could confirm broken drivers:

Files for the P411W-32P are located here:

Support Documents and Downloads

1. The Windows drivers version 2.61.29.00 (dated 06/07/2021) only “work” with firmware version 4.1.3.1 (dated 06/07/2021), if you use them with the Broadcom P411W-32P installed in the system while it has any older firmware installed on it Windows WILL crash during boot with a BSOD even if the HBA isn’t handling the drives Windows itself is installed on.

This is particularly FUNNY since…

2. As mentioned with the latest firmware version 4.1.3.1 (dated 06/07/2021) the P411W-32P cannot detect any devices connected to it (tested U.2 SSDs directly connected to it and the P411W-32P connected to an U.2 NVMe backplane)

Further observations:

The P411W-32P doesn’t seem to need drivers when using it with Windows, you get unkown “Base System Devices” in Windows Device Manager but all features like Sleep (S3, suspend to RAM) work and the performance seems to be the same as with the earlier drivers (version 2.61.19.00, dated 02/25/2021).

It feels (again) like Broadcom is giving me the finger:

• I highly doubt that their support technician actually tested a P411W-32P in their lab with the latest firmware and drivers since these seem to generally break functionality no matter the used OS;

• They stated to having done their testing with “Seagate” SSDs, as mentioned the only Seagate U.2 SSDs I know of are PCIe Gen3 and I explicitly used the Gen4-capable P411W-32P with Gen4 Samsung SSDs;

@wendell

Are you interested in any specific tests?

I saw the PM1733/7.68 TB SSDs reach almost the maximum sequential speeds I had ever previously seen (about 7.350 MB/s instead of 7.400 MB/s).

I’ll do a normie Windows user review with the Icy Dock ToughArmor MB699VP-B V2 backplane.

Any ideas if it is possible to infer PCIe Bus errors indirectly from connected NVMe SSDs’ SMART values?

PCIe Advanced Error Reporting is enabled on the platform I’m currently testing (ASRock X570 Taichi, UEFI 4.80, P411-32P installed in the first PCIe x16 slot, a small dGPU is installed in the chipset x4 slot so the HBA is getting the full PCIe x16 Gen4 interface link).

But I haven’t had a single WHEA ID 17 entry in Windows Event Viewer which I consider highly unlikely in reality since Broadcom’s cables are only available in the absurd 1.0 m length which can’t be good for PCIe Gen4…

Contrary to using NVMe SSDs with a Broadcom HBA 9400 8i8e which is just a black hole for third-party programs it’s nice that CrystalDiskInfo for example can detect the SSDs handled by the P411W-32P just fine

I honestly don’t get Broadcom. Have they been pulling a Microsoft firing their entire QC/QA teams?

[aBav.Normie-Pleb]

@wendell

Good news!

• For the first time ever I can operate my trusty 3 Samsung PM1733 U.2 testing SSDs at the same time on an AM4 system in Gen4 mode

Details on the P411W-32P:

• Windows 10 Enterprise x64 21H2

• Firmware P411W-32P_4_1_2_1_HBA_signed_P14.2.fw (Version 4.1.2.1)

• Drivers: ItSas35_Win10_Win_Server_2016_2019_PCIe_P16 (Version 2.61.19.00)

• It’s a little sad to have one bay of the MB699VP-B V2 empty, I’ll try to at least free some faster Gen3 M.2s to use them in the Icy Dock M.2-U.2 adapter…

• The 3 CrystalDiskMark benchmarks were started at the same time (as fast as I could click and move the mouse);

• Maybe the CPU (3700X) limits performance a bit, if I only launch one CDM benchmark at a time I also get the 7.350 MB/s sequential read just as with the SFF-8654 8i-to-2xU.2 cable directly attached to the SSDs;

• Maybe good news regarding a SFF-8654 8i-to-2xOCuLink cable, the 0.5 m noname one I got from Amazon offers the same performance numbers as the expensive-AF 1.0 m Broadcom cable:

Amazon.com: NFHK PCI-E Ultraport Slimline SAS Slim 4.0 SFF-8654 8i 74pin to Dual...

• I can’t tell if it’s just the P411W-32P that tunes/amplifies the PCIe signals to the max so that shitty cables don’t matter thaaat much or if that cable by chance is actually good;

• Am a little bummed out to not have more Gen4 SSDs to toy with, wanted to get some with some old hardware I had planned to sell but I chose to donate it to a cat café that is hopefully opening up sometime this year;

[gysi]

So downgrading the firmware and windows driver version fixed the problem for you?

[aBav.Normie-Pleb]

Yes,

it was an unfortunate series of events:

• My first experience with PCIe Switch NVMe HBAs was the Delock 90504 that also uses a Broadcom chipset but is older and PCIe Gen3-only: A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, Backplanes, Risers & Extensions - The Good, the Bad & the Ugly - #6 by aBav.Normie-Pleb

• The Delock HBA was buggy AF with its default firmware from manufacturing and finally worked correctly after I bugged Delock’s level 2 support about firmware updates.

• I got the P411W-32P some time before the first suitable cables arrived and I wanted to “prepare” it for smooth operation. It came with non-public early firmware 0.1.0.0 and I thought that it’s going to be the same spiel as with the Delock 90504.

• Broadcom’s support didn’t help, they didn’t suggest to try an earlier firmware version since they “had just tested” it in their “lab” and everything’s fine there.

• But yes, after some distance I think I was stupid not to think of it myself before sending my first unit into RMA.

[gysi]

I’m glad it finally works for you.
I hope I don’t run into similiar issues once my HBA 9500-16i will finally arrive.
I hope the SFF-8654 to U.2 cable ( 05-60005-00 ) will also arrive at the same time.
So the Amazon Cable is compatible/a replacement with/for Cable 05-60001-00 from Broadcom?
Good to know. But for the momment I don’t plan to have the SSD hotswappable.

[aBav.Normie-Pleb]

Thanks!

Yes, that’s the correct (“guaranteed”) cable to connect one of the SFF-8654 8i ports of the Broadcom HBA 9500-16i to two U.2 SSDs.

It’s quite long in regular PC cases, please be gentle (as with all SFF cables) when cable managing it during installation.

I wonder if the 4xU.2 Icy Dock MB699VP-B V2 also works with the 9500-16i, I can feel a slight itch coming…

May I ask why your 9500-16i takes that long to be delivered? Checked locally here (Germany) and it is widely available.

If I don’t get my 9400-8i8e issues properly managed (now that the P411-32P is fine I got the time for them) I’d likely “have” to get 9500-16i as a replacement and adapt two internal to two external ports since for some reason there’s no 9500-8i8e - When getting HBAs new I’d only get PCIe Gen4 ones for more the efficient use of PCIe lanes.

Next I’ll try to somehow scrap together a total of 8 NVMe SSDs to try to reach the mountain top of 32 GB/s with the P411W-32P’s PCIe Gen4 x16 host interface. The available 3 Gen4 Samsung SSDs reach around 21 GB/s sequential read, so the remaining 11 GB/s should be easily doable with 5 PCIe Gen3 NVMe SSDs

Is there a comfortable tool to benchmark several independent drives all at once or is just opening 8 instances of CrytsalDiskMark, selecting a larger test file size and clicking as fast as you can the way to go?

[Log]

@aBav.Normie-Pleb

Is there a comfortable tool to benchmark several independent drives all at once or is just opening 8 instances of CrytsalDiskMark, selecting a larger test file size and clicking as fast as you can the way to go?

I found something interesting here: Flash testing tools - CrystalDiskMark

According to their website, CrystalDiskMark is basically just a front-end to Microsoft Diskspd

The details on exactly what each of these tests does is vague, so I grabbed the command-line it’s passing to diskspd for each of them to see exactly what they are doing. The options passed to Diskspd for each of the tests was :

Test	Read	Write
Seq Q32T1	-b128K -d5 -o32 -t1 -W0 -S -w0	-b128K -d5 -o32 -t1 -W0 -S -w100 -Z128K
4K Q32T1	-b4K -d5 -o32 -t1 -W0 -r -S -w0	-b4K -d5 -o32 -t1 -W0 -r -S -w100 -Z4K
Seq	-b1M -d5 -o1 -t1 -W0 -S -w0	-b1M -d5 -o1 -t1 -W0 -S -w100 -Z1M
4K	-b4K -d5 -o1 -t1 -W0 -r -S -w0	-b4K -d5 -o1 -t1 -W0 -r -S -w100 -Z4K

As a side note, I went ahead and tried CrystalDiskMark in my windows VM:

Lmao thanks ZFS!

[gysi]

“May I ask why your 9500-16i takes that long to be delivered? Checked locally here (Germany) and it is widely available.”

I’m located in Switzerland and ordered it via Digitec. Often products are marked with product available at external distributor, delivered within 3-4 days. When I’ve ordered it in April I think, I got one day later the notification that there’s a delay in delivery and expected delivery is January 2023 :-0
Luckily now the expected delivery date is middle of June.

The same happened with the 05-60005-00 cable. On order page it said 1 piece available at external distributor, delivery in 3-4 days, when I ordered last week. Then after I ordered it, I got a mail that delivery date is unknown. And now when I look into my orders page it also says expected delivery date also middle of June.

At least I’ve got now the 05-60006-00 cable which I also ordered when I ordered the 9500-16i.

Currently I don’t plan to use any Icy Dock enclosures, as I don’t have an (external accessible) 5,25 bay.

[aBav.Normie-Pleb]

@gysi

My condolences regarding the waiting period.

Funny coincidence, have a pleasant memory of digitec’s Zürich location, picked up my main living-room speakers there back in 2007 that I’m still using to this day

@Log

Thanks for the hints.

Just scraped 6 SSDs together, think I can get 8 soon and I think the poor 3700X is bottlenecking the performance, but no stability issues at all, even when going to Sleep/S3 while benchmarks are writing or reading from the SSDs connected to the P411W-32P

Hope I can test the P411W-32P maxed out in combination with a 5950X.

@wendell

I think I encountered a hardware bug with the Icy Dock ToughArmor MB699VP-B V2 - can you confirm the following?

When looking at it horizontally the status LED of two bays that are located over each other seem to be activated in parallel.

Example: The drive in the top left bay gets accessed and that one’s and the status LED of the bottom left also flashes, even if there is 100 % no access to that drive. The same pattern can be observed on the opposite side.

Don’t like such strange things

[aBav.Normie-Pleb]

@Illumous

If you’re using Windows with a system that has the P411W-32P installed in can you confirm my Bluescreen observation when using the latest drivers version 2.61.29.00 (ItSas35_Win10_Win_Server_2016_2019_PCIe_P18_0) with the HBA having any older firmware up to 4.1.2.1 (P411W-32P_4_1_2_1_HBA_signed_P14.2.fw)?

Only in combination with the latest firmware version 4.1.3.1 (P411W-32P_4_1_3_1_HBA_signed_P14.3.fw) the driver version 2.61.29.00 (ItSas35_Win10_Win_Server_2016_2019_PCIe_P18_0) doesn’t cause a Bluescreen for me on my two tested platforms (but then again, no SSDs are showing up at all with that latest firmware).

I’m trying to collect as much details as possible to maybe be able to resolve this in concert with Broadcom’s tech support.

[Illumous]

Well… A lot has happened since I’ve last been here. I’ve had to perform minor data recovery (Windows ate dirt), troubleshoot hardware (H/W is fine at least XD), and reinstall Windows… It’s been fun.

For the P411W-32P:

1. There are several versions of drivers depending on which version of Windows is installed. And then, there are two parts to the driver package. One half appears to be for the management endpoint (ItSas35.inf and ItSas35.sys) and the other is for the hot swapping mechanism (lsinodrv.inf).
   I tend to have issues with the first half - ItSas35.inf and ItSas35.sys. I have not tried previous versions of this file. (I’m still running firmware version 4.1.2.1) However, any attempt to install the driver on the last couple of versions of Windows I’ve had (1809, 20h2, 21h2) promptly results in a blue screen of death. If left alone, Windows will eventually install a version that does not cause crashes. Something to note, I have been using a third party utility to manage drivers.

2. It is on my to-do list to eventually try the latest firmware and driver combination. I honestly do not want to break a working configuration quite yet.

3. The driver read-me does warn about crashes if a particular patch (KB4482887) isn’t applied. However, Windows version 1809 and newer already has this patch applied. Weird…

4. I am running mine in PCIe 4.0 x8 and have noticed that I may have hit a bottleneck of about 14 ish GB/s with 3x Samsung 980 Pro 2TB SSDs. Damn 3080 consuming lanes… =P @aBav.Normie-Pleb, what do you peak at in a PCIe 4 x8 config?

@aBav.Normie-Pleb

I’m trying to collect as much details as possible to maybe be able to resolve this in concert with Broadcom’s tech support.

I’d love to know what exactly their test rig is composed of…
Also, thank you for helping out with this.

@aBav.Normie-Pleb

In The guide for the Tri-Mode Adapters ( https://docs.broadcom.com/doc/MR-TM-StorCLI-UG - only found one for the HW RAID variant but firmware stuff should be similar) when looking at erasing parts there is a mentioning of erase exceptions for a “manufacturing data region”, do you have an idea what this is about? So there are 4 different firmware parts?

The guide mentioned (just skimmed it) looks to be applicable for your use case.
I don’t remember off the top of my head what Manufacturing Data Region is. I’ll have to wait for when brain.exe decides to resume operation. Until then, you might want to consult someone who is more versed. In the meantime, play it safe and don’t nuke that portion.

Side thought, @aBav.Normie-Pleb

@aBav.Normie-Pleb

Platform A: ASUS ProArt B550 CREATOR/5750G/128 GB ECC Micron; HBA in first x16 PCIe slot (running at PCIe 3.0 x16);

Have you been able to verify ECC functionality for that CPU/mobo combo? The APU in particular.

[Exard3k]

@Illumous

Have you been able to verify ECC functionality for that CPU/mobo combo? The APU in particular.

ECC isn’t supported with APUs (particularly because the shared memory of the graphics unit doesn’t like ECC). AMDs workaround for this are the Pro versions of Ryzen (very rare in retail as they are mostly an OEM thing).

[aBav.Normie-Pleb]

@Illumous

Have you been able to verify ECC functionality for that CPU/mobo combo? The APU in particular.

In Windows via cmd.exe:

wmic memphysical get memoryerrorcorrection

Meaning of the values:

0 (0x0) Reserved

1 (0x1) Other

2 (0x2) Unknown

3 (0x3) None

4 (0x4) Parity

5 (0x5) Single-bit ECC

6 (0x6) Multi-bit ECC

7 (0x7) CRC

Ryzen does Multi-bit ECC (6). As for testing ECC with software the easiest way as far as I know is PassMark Memtest that supports error injection.

@Exard3k

ECC isn’t supported with APUs (particularly because the shared memory of the graphics unit doesn’t like ECC). AMDs workaround for this are the Pro versions of Ryzen (very rare in retail as they are mostly an OEM thing).

Yes, that’s why the 5750G I’m using is such a PRO SKU.

@Illumous

Thank you very much for your response!

Could you give complete CPU/motherboard/memory/UEFI version details of the systems you have tested your P411W-32P with?

[Illumous]

@aBav.Normie-Pleb

Yes, that’s why the 5750G I’m using is such a PRO SKU.

Ah! That explains everything. Thank you.

@aBav.Normie-Pleb

Could you give complete CPU/motherboard/memory/UEFI version details of the systems you have tested your P411W-32P with?

I can give the appropriate details potentially this weekend. I otherwise do not have an available time slot.