A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

No, parts with PCIe Gen4 ReDrivers are available (for example various adapters sold by MicroSATACables).

I had a little bad luck with a few units arriving DOA (M.2-to-SFF-8654), but that can happen to any manufacturer when looking at how packages are handled.

I just tested the Micro SATA Cables M.2-to-OCuLink adapter with a redriver and a 50cm active OCuLink cable. There are less WHEA events, but still I got 6 in just a minute of random reads. Turning on the motherboard’s special redriver settings lowered the number of events by about a half (I was no longer getting them when the drive was not under load), but they were still there.

I even purchased the 75cm and 100cm ones (also claimed to be PCIe 4.0-ready) for testing in case the 50cm one worked with the redriver, but looks like even 50cm is too long of a length. I’m starting to think PCIe 4.0 over a cable is just not meant to be.

But don’t take my word for it. I’m just a newcomer to this stuff. Still hoping someone finds a workable and reproducible solution and shares it. :slightly_smiling_face:

EDIT: Out of curiosity, I also tested the 75cm cable. 16 WHEA events in less than a minute. 100cm is out of the question. Supermicro sells a 37cm cable (not active AFAIK). Not sure if I want to buy one just to test the lengths in between 25cm and 50cm.

1 Like

ohhh no~ help~after the step erase my p411w show the yellow led(system error ) and i can`t find any device in g4xflash tool…what happened!!!
is there any way to save my card byself?


help me pls!!! my p411w show system erro led .an can`t detect any device in windows and efi os . :sob: :sob:

That’s something most likely only Broadcom’s Tech Support can help with.

Does the P411W-32P still show up itself as PCIe Devices when looking through the system’s topology with something like HWiNFO?

(Edit due to confusion about Broadcom’s version naming scheme)

bad news.no any info about p411w 32p in HWiNFO.

:sob: :sob: :sob: :sob: :sob:

I found a combination that works. No WHEA events logged and PCIe 4.0 speeds!

Looking back, this should have been so obvious. I had already bought their adapter first thing, so why not their cables—which explicitly advertises PCIe 4.0 support—too?

1 Like

I decided to reach out to Broadcom over the weekend regarding the missing drives issue when on the latest firmware.
Here’s a screen cap of the email I received.

Just sent an email requesting the beta too.
I’ll try to post it when I receive it.

What I originally sent them:

This topic is in regards to P411W-32P

On initial 0.1.0.0 firmware, everything shows up. However, upon updating to the latest (v4.1.3.1 - Shows up as v1.3.1), absolutely nothing will show up regardless of motherboard (Supermicro, Asus, etc…) or settings. HBA shows up in both Linux and Windows though.

With the previous update (v4.1.2.1 - shows up as v0.1.2.0), everything shows up and functions as it should.

A variety of systems and SSDs by various manufacturers were tested to no avail. Different cables were tried too. I have heard Seagate SSDs work, but have not been able to test that particular variable. I’ve personally tried SSDs by WD, Samsung, and MSI.

The latest firmware hates Samsung SSDs. Nothing shows up regardless of OS or motherboard firmware. It plays a little bit nicer with WD SSDs and will at least show the (WD SanDisk) NVMe controller when attached, but still no storage medium.

I have been working with another entity who suffers from the same issues. A new card does not resolve the issue.

2 Likes

Well,well, well…

“We are aware of the issue and should have a fix posted shortly.”

Broadcom Support, May 2022:

“Dear Mr. Normie-Pleb, we’ve tested your claims in our lab and everything works as it should. You must be the problem!”

4 Likes

Tsk tsk tsk… At least it’s getting somewhere now.

Just heard back:
image

Anything I should add (or ask)?

1 Like

Maybe poor Joseph got reprimanded for sending something out in writing that their products might be less than perfect?

@Illumous

Have you been using an “UMB Tri-mode backplane”?

Heh! Funny… /s
Nope. I’m not that rich.
I’ll ask about their setup though.

What SSDs, SSD form factors, and methods have been tried here thus far?

When asking about that I only got that response with that obscure Seagate PCIe Gen3 U.2 SSDs no one in their right mind would intentionally use with that PCIe Switch.

Some people… :person_facepalming:

What SSD formfactors and connection methods have been tried so far?
M.2, U.2/U.3, EDSF, enclosures, pcie add-in cards…?

Good luck getting these details, I almost asked them to send me a video of their test lab where you could see the display, display cable going to the system, the motherboard and all SSDs in the same shot while they perform all the tests that went so without any issues - but that might be perceived as a micro-aggression :wink:

1 Like

Ooof! I’ll give it a wallop. :grin: :smiling_imp:

I mean I really haven’t seen ANY system where these things didn’t happen:

Does the same company that organizes Pfizer’s and Moderna’s drug trials run Broadcom’s engineering lab?
(SCNR)

For easier reading here’s a list of everything I’ve looked at with the P411W-32P.

No hardware configuration change changed anything in the pattern of issues experienced.

  • M.2 SSDs were used within an Icy Dock EZConvert MB705M2P-B M.2-to-U.2 adapter.

  • U.2 SSDs were tested directly connected via cable (Broadcom 05-60005-00) or Icy Dock 4 x U.2 backplanes MB699VP-B and MB699VP-B V2 via Broadcom 05-60002-00 (SlimSAS HD) respectively 05-60001-00 (OCuLink)

Tested SSD list:

  • Intel Optane 905P 480 GB (U.2, PCIe 3.0 x4)
  • Intel Optane P4801X 100 GB (U.2, PCIe 3.0 x4)
  • Micron 7450 3.84 TB (U.3, PCIe 4.0 x4)
  • Samsung 960 PRO 1 TB (M.2, PCIe 3.0 x4)
  • Samsung 980 PRO 2 TB (M.2, PCIe 4.0 x4)
  • Samsung PM1733 7.68 TB (U.2, PCIe 4.0 x4)
  • WD SN550 2 TB (M.2, PCIe 3.0 x4)
  • WD SN750 2 TB (M.2, PCIe 3.0 x4)
  • WD SN770 2 TB (M.2, PCIe 4.0 x4)
  • WD Ultrastar DC SN630 7.68 TB (U.2, PCIe 3.0 x4)
1 Like

I wonder what the signal insertion loss for such a set-up is. You’ve got 4 connections in total for a drive to connect to the card and then some distance to cover:

  1. the M.2 drive to the adapter
  2. the adapter to the enclosure
  3. the enclosure to the cable, which is a meter long
  4. the cable to the Broadcom card

Do the PCIe 4.0 drives perform at PCIe 4.0 speeds without errors?

No change directly connected to the P411W-32P via cable or in the backplane. I’m sure there are PCIe Bus Errors but no way of checking them with the P411W-32P.

I don’t have a way to verify this but I’m almost certain that Broadcom is using a good amount of active PCIe redrivers with their absurdly long 1 m cables.

I only used them so that Broadcom Support couldn’t say I used some unsupported third-party cables. I also tested those, nothing changed (Even though BSoD crashes also happened without any cable connected to the P411W-32P).

I’ve tested 4 x Samsung 980 PROs in an ASUS PCIe x16-to-4xM.2 PCIe Gen4 Bifurcation AIC and that worked without any PCIe Bus Errors on a system where PCIe Advanced Error Reporting was working, as far as I know that card is passive without any active signal processing.

1 Like