A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

Broadcom just keeps on never not disappointing:

  • Their drivers’ signatures are still broken after over a year:

  • Currently cannot even boot up Windows 11 22H2 (Windows Update level up to present, all third-party drivers up-to-date including AMD’s new August 2023 chipset drivers), the P411W-32P causes a Bluescreen even with Windows’ default drivers:

No cables are attached to the P411W-32P

  • But Broadcom JUST KNOWS what the really important fixes are:

Can anyone describe in a pleb-friendly how to upgrade the P411W-32P’s firmware in a Linux or EFI environment? Have never done that before :frowning:

Broadcom isn’t even once checking what they are releasing, they just squirt out one runny turd after another. :face_with_symbols_over_mouth:

@wendell

Another pretty please for a dedicated video just shaming Broadcom all around.

4 Likes

PCIe signals are not effective when driven out of the mobo interface (PCIe slots) - that being said extender cables have been in use for long - however they do introduce performance and irregular heat dissipation issues. The performance loss may not be too noticeable but if you are using GPUs or video cards, it will be more obvious - regardless the margin of error is low. The cables must also meet the power connectivity requirements of the system in which it is to be installed. And cost - high quality cables are expensive!

I am baffled!

1 Like

Me, too!

It’s like a special Ed class made a $ 100 Billion start-up and no one seems to care.

Mostly it’s just low latency/high random IOPs for these high performance applications that *most software RAID doesn’t fulfil. There are a couple software “raid” solutions which buck the trend of poor performance like xinnor, but it has other drawbacks.

Modern RAID cards can do anything the switch adapters can as I understand it, including passthrough and using internal pcie switch to adjust NVME link width. I should probably confirm NVME passthrough on my RAID card before saying that though.

I’d like to see CXL implemented in future RAID cards to further improve performance and scalability. We really need accerators for storage now more than ever, CPUs just can’t keep up with the fire hose of data coming into them efficiently.
I’m not sure how Intel’s DSA accerator factors into this paradigm though, perhaps for Intel systems running large amounts of NVME, the traditional RAID card won’t be as useful. But AMD doesn’t have anything like DSA.
​​​ ​ ​

​​​ ​ ​

@aBav.Normie-Pleb
Did Broadcom support mention virtual pin port mapping functionality depreciation in any of their responses to you earlier?

2 Likes

I also tried updating the latest firmware.
Unfortunately, after the update I didn’t see my hard drives in the motherboard bios in NVMe devices. (The hard disks are visible with a working firmware)


(This is normal firmware.)

The latest drivers I also tried, only the drivers in the “Win2019_Server_RS5_LTSC_f18_x64” and “Win2022_Server_x64” folders were able to be installed correctly, but due to the firmware this still didn’t help.

I’m currently using win11 22h2 with an intel cpu and z790 motherboard, and I’ve installed all the drivers in the “C:\Windows\System32\drivers” and “C:\Windows\System32\DriverStore\FileRepository” folders for itsas35i. syts. sys, Mpi2Ses.sys" and “C:\Windows\System32\DriverStore\FileRepository”.
After that, reinstall the normal version of the driver “itsas35.sys” and it will work normally.

3 Likes

Your firmware experience indicates that Broadcom hasn’t backed down from removing the support for directly attached SSDs which they had killed in the previous firmware release P14.3 on purpose so the HBA basically only ever works now when you use it with a specific UMB backplane model line.

Sucks that there isn’t any reflection on their actions :frowning:

1 Like

Is this the correct nomenclature for saying directly attached SSDs won’t work anymore?

If so, that’s “old news”, we wondered about that in this this thread last year or so when testing the firmware package P14.3 for the P411W-32P where suddently no SSDs were detected anymore.

Initially we thought it to be a bug (because it would be so stupid) but Broadcom support confirmed it to be intended behavior. If you want to use firmware P14.3 or newer you need to connect the HBA to an UMB backplane:

I’m a bit confused by this: itsas35.sys is part of the the drivers for the Tri-Mode HBAs like the 9500-16i, this file isn’t present in any of the driver folders for the P411W-32P, here’s their content:

Am I having a Biden moment?

lack of “native” VPP support could cause the issue, but its just a guess on my part as to the cause of the issue. If broadcom decided to rely solely on the backplane for the pin reconfigurations it would explain… I’m just trying to rationalize why they would drop this support

1 Like

You’ll be driven into an asylum!

My subjective gut feeling regarding Broadcom’s shiffty behavior:

They found unfixable hardware issues (in all their current designs dating back to the 9400 model line) so they’ve decided to put their heads in sand until the warranty periods for most of the affected produced units is over so that f’d over customers can’t demand refunds.

I’ve personally never witnessed a reputable company cutting advertized features in the middle of a product’s life cycle like Broadcom has done.

5 Likes

hmmmm……itsas35.sys is the driver provided with P14.2. (I’m using the program under win11.)
This is actually the ItSas35_Win10_Win_Server_2016_2019_PCIe_P16 driver downloaded from the official website.

The driver files since the P23 are all mpi2ses.sys, I did use the 9500-16i driver which is itsas35i.sys not itsas35.sys.
Both of these drivers and the P411W-32P_4_1_2_1_HBA_signed_P14.2.fw firmware will cause a BSOD.

It occurs to me that after updating the firmware I’ve been using icydock’s MB699VP-B V3,No try sff-8654 to sff-8639 connector…
Maybe it’s because of the backplane?

2 Likes

Known issues:

  • P411W-32P: Firmware package P14.3 or newer: HBA will no longer detect SSDs IF you don’t use an UMB Tri-Mode Backplane.

  • Using drivers older or newer compared to a firmware package version will likely cause a BSoD during the boot sequence.

  • What firmware and driver combination actually worked for you?

@wendell

Level1Store suggestion: A “real” UMB-compatible Tri-Mode U.3 backplane (4 x 2.5" in a 5.25" form factor) that lets you use SATA, SAS and NVMe SSDs for maximum flexibilty?

2 Likes

My google-fu brings up Evolving Storage with SFF-TA-1001 (U.3) Universal Drive Bays - StorageReview.com

Reads like a laudable effort towards standardization. But your reports on the current state seems to be botched.

I suppose the drives (SATA, SAS, NVMe) expect a certain protocol and therefore electrical wiring, the backplane supposedly has the capability to differentiate, and for whatever reason (sell two products instead of one?) the controller does not have that capability or it was dropped.

1 Like

That’s the beautiful absurdity making me want to throw all stuff with Broadcom on it out the window.

The P411W-32P is a pure active PCIe Switch HBA, meaning you absolutely can only ever use it with PCIe NVMe SSDs and nothing else.

Them stating you intentionally only can use it with a Tri-Mode UMB backplane (to differentiate between NVMe, SAS and SATA drives being plugged in) from now on and how dare you demand bug fixes anyway is almost making me doubt my sanity.

PS: The IcyDock V2/V3 U.2 backplanes that are labeled “Tri-Mode compatible” are a similar case: It only means that you can connect them to Tri-Mode-capable HBAs like the 9500-16i for example, NOT that you can actually use it as a Tri-Mode backplane with SAS and SATA as well as the obvious NVMe SSDs.

2 Likes

So P411W-32P documentation indicates VPP support but this is dependant on the backplane (?).

The p411w is a switch adapter solely for NVMe - tri mode connectivity is not available for this class of HBAs - it uses a completely different controller chipset.
The backplane support is transitioning from SES/SGPIO and VPP - to UBM. However, the adoption is slow - VPP is more legacy and SES/SGPIO is still being used from what I have heard.

1 Like

We need a forum wiki you manage with know issues on these devices

5 Likes

Hello, does anybody of you has some information about NVMe hot swapping?
I am currently building a setup that I can use to transfer big amounts of data between multiple U.2 SSDs and had the problem that my PCs dont work by using a adapter from NVMe to MINI-SAS HD to connect it directly to my ICY Dock with MINI-SAS HD conenctors. It should be possible to swap the U.2 SSDs while the PC is on but I have the problem that the pc dont recognize if the SSDs are connected or not, only after I try to transfer Data it gets a error massage.

Hey there! I’m terribly sorry, if the question(s) are already answered, but search(phrases) didn’t conclude my particular question.

I made a deal on an Gigabyte MJ11-EC1, and loocking for a “budget-sane” solution.
Starting point:

  • Using the SATA-Ports for storage
  • M.2 (3.0x4) is intended for a 10GBE via an open-end X4 Slot-Adapter
  • SlimSAS 4i (3.0x4) is configurable to x4 PCIe or SATA x4
    (It looks to me like a MiniSAS???)
  • SlimSAS 8i (SFF-8654 8i)

I’d want to attach an ARC A380 (or A310 if I can get my hands on one) for Video-Transcoding, additionally, I’d like to attach 2 NVME-Drives (depending on the solutions $$$).

  • What would be the best bang/buck solution to get a “second” PCIe slot (M.2 counts as my first one)? The SlimSAS 8i to PCIe x8 (electrical) or SlimSAS 4i to PCIe x4 (electrical) and a SlimSAS 8i → 2x U.2 Cable-Solution?

  • Starting of my research I’m roughly around 100€ to get some PCIe Cabling and Slots from SlimSAS PCIe gen4 Device Adapter 8i to x4x4 - 4W – C-Payne PCB Design which is somewhat around the same money I spend for the server :see_no_evil:
    Are there better (cheaper & working) solutions?

Thanks for hanging on through my scrabled thoughts :sweat_smile:

Scratching my thoughts of being a cheapskate in the wrong kind of way:
The C-Payne stuff seems to be positively reviewed, cheaping out won’t do me any good… Will go with the bifurcated x4 x4, which will be plenty bandwidth for my tasks of transcoding, or whatever I’ll throw at it!