A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

No problem at all … I created a txt file in the hard drive connected to the P411w-32p, went back into the system after S3 sleep and modified the txt, the file was modified normally, no problems with unreadable, file hard drive does not exist, BSOD.
(The hard drives I use are intel’s P4610, P5510 and Samsung’s PM1735, all without problems)

1 Like

Welp! I had played around with the 2023-03 update of Windows 11 22H2 and my P411w-32p. Fresh install, no internet, and at first, all I was getting out of the ordeal were BSODs.
A word of caution to anyone who wants to try this in the future…
The driver available out of box will cause BSODs with firmware version 4.1.2.1.

My solution was to boot without the card, rename ltSas35i.sys in C:\Windows\System32\ to ltSas35i.sys.ori

Then reboot with the card, rename ltSas35i.sys.ori back to ltSas35i.sys, and disable the “Broadcom PCle Switch Management Port PEX880xx” in devmgmt.msc


Worked like a charm afterwards. No need to tinker with the firmware.
I’m definitely avoiding W11. That was awkward and painful af!

1 Like

From the P411W-32P user guide:


@NeoThermic, There’s a summary at post 193.

Just to be sure: That changed txt file remained changed even after a reboot of the system? I’m asking since Windows’ memory write cache could lead to a file seemingly being changed even without it actually being changed on the drive.

I’m not trying to sound too hyperbolic but you are the first user EVER I’ve seen outside Broadcom’s “engineering lab” that has seen S3 and waking up working with the P411W-32P. It happens on all of my systems, even if no SSDs or cables are connected to the HBA.

Could you please give a complete report of your hardware configuration including motherboard BIOS version etc. (memory amount etc.)?

I don’t want to experience a new hope only to get hurt again eventually :wink:

Guess I’ll also take a look at Windows 11 now.

I opened the txt again today after restarting the computer and took a look at it, and it is indeed the correct modified text …
However, I am using a Chinese system, although I checked the state of S3 sleep is indeed enabled, can not fully guarantee the difference between the region and the system version.
Device information from intel Extreme Tuning Utility program (intel XTU)
System version is win11 22H2 22621.1485 Professional


1 Like

Thanks for these additional details!

I have been following through this thread, and I wanted to see what people were recommending right now for generic cables to connect SFF-8654 to U.2 at Gen 4 speeds. It seems a lot of folks were recommending Broadcom 05-60005-00, but I wanted to ensure that this wasn’t just with the P411W-32P that everyone seems to be using. I would be using this on an Rome2D16-2T with onboard SFF-8654 ports. Thanks all.

Unfortunately to the best of my knowledge no one here could validate a solution with solely passive SFF-8654-8i PCIe Gen4 cables that didn’t rely on PCIe ReDrivers.

Youre the Normie-Pleb wendell was mentioning in the video! that is a real shame that redrivers were needed. What about m.2 to U.2 short <10cm runs? Any. luck there?

Yes, I’m that Normie-Pleb that has been annoying Wendell for a few years with the bugs I encounter.

For M.2-to-U.2 the combination Wendell is showing off in that video seems to be reliable:

Generally yes, the shorter the cables are the better but on the other hand cables shorter than 50 cm are less common and might be manufactured in more “flexible” factories that could need better quality control.

1 Like

Has everyone that has encountered trouble with pcie cabling been using 85 ohm cabling?

The pcie spec targets 85 ohm impedance for pcb traces, but in reality the mating connector (SFF-8654 or OCuLink) will drive up true impedance quite a bit. More signal power may be able to be delivered to end devices using the SAS-spec 100 ohm cabling.

1 Like

I did check for that (printing on the cables and if available spec sheets for the complete cable).

But of course, ink is cheap and I don’t have the equipment to actually measure the signals over these cables…

Ironically a 50 cm SFF-8643-SFF-8643 SAS3 cable that came with an Intel SAS3 expander about ten years ago was the thing that worked the very best in a completely passive chain with PCIe Gen4 with an M.2-SFF-8643 adapter and the first-gen of Icy Dock’s 4 x U.2 backplane.

I will have the 5 cm and 20 cm ones to test in about a week’s time. Shipping from China takes a while.

I have the 15 cm ones already and they work fine:

Summary
2 Likes

Thanks all.

Is that a single-piece solution M.2-to-SFF-8639?

That should be better for signal integrity since there aren’t unnecessary connector-plug transitions.

Edit: Yes, just saw your previous posting with the eBay link:

Yes. You cannot take it apart. And I’ve found these things to be very inflexible despite appearances. If you order them and have some change to spare, I would suggest making very careful measurements and ordering a size shorter and/or a size longer. The 15 cm one I have blocks my second PCIe 5.0 slot and I cannot bend the ribbon in any way to unblock the slot. Hence, I ordered an even shorter one to replace it.

1 Like

I wonder if any were able to change the firmware, set a mix config on those pex switch chip ? I see the new P411W-32P that can be handy to have different speed, but also have firm issues. But just doing a reconfig on the popular pex8749 seem impossible. Like having the port (as show in plxmon) 11/10 & 16/17 set in 8x mode and the rest at 4x. Or finding the support or prod manual look impossible.

Maybe this could also help with getting rid of PCIe issues?

Some might have heard that there have been fu…ups in AMD’s AM5 platform leading to catastrophic sudden failure of Ryzen 7000 CPUs and some motherboards:

One facet here (but not the only one) was some motherboard manufacturers (ASUS being the worst) blasting the SoC voltage to brute-force better memory compatibility with the IO die.

That same IO die isn’t only handling the memory but also all PCIe devices.

Has anyone tested if reducing the SoC voltage on AM4/AM5 leads to more PCIe errors and increasing it might reduce them?

A question for all. I got one of those PCIE gen 4 M.2 to U.2 adapters, 15cm and I am pretty sure I am seeing errors with the drive hooked up (not causing problems in the VM).

[ 5611.588482] nvme 0000:02:00.0: AER: aer_status: 0x00000081, aer_mask: 0x00000000
[ 5611.589112] nvme 0000:02:00.0: [ 0] RxErr (First)
[ 5611.589113] nvme 0000:02:00.0: [ 7] BadDLLP
[ 5611.589114] nvme 0000:02:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
[ 5611.589765] nvme 0000:02:00.0: AER: aer_status: 0x00000001, aer_mask: 0x00000000
[ 5611.590399] nvme 0000:02:00.0: [ 0] RxErr (First)
[ 5611.590401] nvme 0000:02:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
[ 5611.591035] nvme 0000:02:00.0: AER: aer_status: 0x00000001, aer_mask: 0x00000000
[ 5611.591662] nvme 0000:02:00.0: [ 0] RxErr (First)
[ 5611.591663] nvme 0000:02:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID
[ 5611.592295] nvme 0000:02:00.0: AER: aer_status: 0x00000001, aer_mask: 0x00000000
[ 5611.592919] nvme 0000:02:00.0: [ 0] RxErr (First)
[ 5611.592920] nvme 0000:02:00.0: AER: aer_layer=Physical Layer, aer_agent=Receiver ID

I presume these are the errors that everyone has been having with unsupported connections?

Thanks

PCI 0000:02:00.0 is my Samsung PM1733

Were you able to do a firmware update your PM1733?

I’ve got some 15.36GTB drives that are driving me up a wall trying to get to work with ESXi. Can’t figure out why they are not able to be seen.