A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

No, but it was seen by proxmox and truenas so that worked out for me.

Me neither, firmware files at hand but the public Samsung tools don’t recognize my OEM PM1733 in the system (PCIe directly from CPU).

Can I use SlimSAS 4i cables in 8i connector? Bit off topic because I’d only be using SATA connections. My MB has 8i connector (S/M H12SSL-i) and I’ve got spare 4i to 4 SATA cable but I haven’t mustered the courage to try (read: I don’t want to break uptime to try)

Have you tried to secure erase them? When I have used the secure erase utility embedded in my asus mobo, I’m often presented with an option to format the drive with either 512 byte or 4k sectors (depends on the model).
I seem to recall ESXi not playing nice when the the destination media had 4k sectors.

It’s my understanding that the 8i is physically wider than the 4i. You’ll likely need the appropriate cables.

This alone wouldn’t be a problem. USB 3.0 initially had a backwards-compatible micro connector that was wider than its predecessor. But SlimSAS 8i and 4i differ by more than width. The cable locking mechanisms are centered with respect to the pins underneath, so inserting a 4i cable into an 8i port connects the wrong pins.

3 Likes

I concur. The aforementioned logic is sound.

I was reading through but couldn’t find whether someone had gotten PCIe 4.0 drives to work on the MB699VP-B V1 without bus errors. I have the Linkreal SlimSAS cards with and without a ReTimer coming in the mail. Will that be enough?

The MB699VP-B V2/V3 is difficult to source in Australia at a reasonable price or even at all. TechBuy.com.au only has the MB699VP-B V1 yet has the M.2 version of MB720MK-B V2. I have emailed them whether they can source the MB699VP-B V2 and am waiting on a response.

Any other Aussies here?

I wasn’t able to get the V1 backplane with PCIe Gen4 without any errors, Gen3 works without issues. The best SFF-8643 cable (meaning the least amount of errors) was an old SAS3 cable that came bundled with an Intel SAS3 expander, even though it was “just” a SAS3 cable.

Every other cable was worse, even the ones claiming to do PCIe Gen4.

That’s unfortunate. Was that even with a ReTimer? I’m going to try the Linkreal SlimSAS PCIe 4.0 ReTimer.

No, that try was with a completely passive chain. If you have luck with ReTimers on a system where you know that PCIe Advanced Error Reporting is actually working please share your experiences here!

While I’m getting a PCIe ReTimer, cables and SSD. I won’t personally be buying a PCIe 4.0 capable system as the current server works ok for the time being. Just getting ready for a later upgrade.

I will try to ask a couple friends with newer gaming PCs if I can test the chain on their system and report back here.

Also weren’t you the one trying a Broadcom Tri-Mode HBA? That’s technically a kind of “Re-Timer” as it’s outputting an entirely new signal from the HBA itself.

  • The Broadcom PCIe Switch HBA (P411W-32P) and Tri-Mode HBAs (9400-8i8e and 9500-16i) mask PCIe Bus Errors, so you’re basically flying blind. The Icy Dock V1 U.2 backplane also seemingly “works” with PCIe Gen4 over SFF-8643 cables in a completely passive adapter chain (for example from M.2 slots or PCIe Bifurcation adapter AICs) if you put your head in sand and are willing to ignore PCIe Bus Errors

  • I have trust issues with Broadcom, their HBAs only function with specific firmware versions and if you don’t use specific driver versions with them they’ll cause BSoDs on Windows during boot

  • They only function correctly if you don’t use system sleep (S3), which they supported in the past and officially still do. I personally couldn’t find a single system where it properly works in the present: After waking a system up from S3 sleep it crashes with a different BSoD after working seemingly okay for a short while

  • Broadcom’s support is BS-ing you and stringing you along instead of trying to actually fix the issues.

  • I’m personally very pissed about this since I wanted to use the P411W-32P in a Windows system with Icy Dock V2/V3 backplanes that is intended to go to sleep regularly

  • Without Broadcom HBAs my systems can use S3 sleep with 0 instability issues

2 Likes

So, it seems these adapters are hit-or-miss. I had ordered a 5 centimeter and 20 centimeter version from a different seller for testing. (The one who sold me the 15 centimeter one only had two sizes to choose from.) Both the 5 centimeter and 20 centimeter adapters I ordered could not carry a clean PCIe 4.0 signal.

The 5 centimeter adapter could at least transfer data, albeit a hundred or so megabyte per second slower than the 25 centimeter OCuLink solution and giving me an error every second.

The 20 centimeter one outright made my system unusable, with my screen flooding with these PCIe errors.

2 Likes

Hello everyone, I’m new here.

I just built a brand new “consumer grade” desktops under Windows 11 x64 - because for the coming >1y this will be used as my primary and sole computer. (Unraid in the future by then).

This build is an standard consumer desktop parts, AMD X670E / 7900 (65w) etc… 100% of the parts are brand new.

And obviously a brand new Broadcom HBA 9500-16i !

I confirm I’m experiencing BSOD since I built that system. (and shutdown issues with the original Win11 drivers and the first original factory “P14” fw and bios version from 2020 → seems gone now…)

This is the list of all the drivers I tested, from the built-in “WHQL” version in Windows 11 22h2 and later versions provided on Broadcom website. All of these seem to give the same results - BSOD coming back from sleep mode. What a bummer because I want this desktop to go in sleep mode as it’s not supposed to stay 24/24, at least for the time it’s my main gear under Windows 11…

• Win 11 22H2 x64 - 2.61.29.80 - factory FW and factory BIOS
• Win 11 22H2 x64 - 2.61.58.00 - last FW and last BIOS version
• Win 11 22H2 x64 - 2.61.54.00 - last FW and last BIOS version
• Win 11 22H2 x64 - 2.61.48.00 - last FW and last BIOS version
• Win 11 22H2 x64 - 2.61.45.00 - last FW and last BIOS version
• Win 11 22H2 x64 - 2.61.41.00 - last FW and last BIOS version

It still BSODing my system with all the drivers “up-to-date”. Windbg links it everytime to ItSas35.sys which this HBA driver.

I’m in contact with Broadcom support by email and sent them 15x minidumps (Win dumps) and 6x “lsiget” logs to them via their ftp (1st batch one week ago and the rest few minutes ago).

I’m sad to discover this thread and thanks to LiKenun and the others here for all the messages and investigation (hasn’t read all yet).

So apparently it’s an issue on-going since over one year and not card/drivers specific which worries me a bit it hasn’t been solved since.

Did you all contact Broadcom support and send them your minidumps and lsiget logs and more ?

Rgds,

Spinmoon

1 Like

I also have two issues, if anyone could confirm the same behavior or not please,

  1. Now that I upgraded my HBA firmware and BIOS : Both upgraded with STORCLI with 9500_16i_Pkg_P26_MIXED_FW_BIOS_UEFI

I can’t check fw/bios version anymore or any information reported by STORCLI after having upgraded because STORCLI doesn’t see my card anymore… (“StorCLI Controller 0 not found” - even with admin rights - same issue via Ubuntu WSL) :-/

Note : I reported this info few days ago to Broadcom and seems then they removed P26 fw/bios from the download page, they only list “archive” older versions for fw/bios now… it’s curious!

  1. Issue which is also new and I don’t where it comes from, CrystakDiskInfo 8.17.16 x64 doesn’t report my 10x HDD anymore.

It worked few days ago, I took screenshots of 7x disks SMART info, now in-between I upgraded the HBA fw/BIOS/drivers and added 3x more HDD but impossible for CrystalDiskInfo to show me those anymore, still showing 2x SSD (via NVME M.2 MB slots though). It’s strange because I change nothing to my computer except adding 3x new hdd (3 on 10 were DOA so sent them back few days ago). It’s a shame cause this is the defacto SMART tool.

Putting back original Win 11 drivers, no change.

Hard Disk Sentinel 6.10 is showing me my disks and SMART info fine. Not sure what’s happening with CrystalDiskInfo…

1 Like

I’m not sure why you think sleep mode will/should be supported on “server” hardware? If you want that you likely need to look for a AHCI adapter instead such as the ASM1166 or similar for it to work somewhat reliable.

Is it too much to ask for a close to $1’000 HBA card with official support for Windows 10/11 to not BSOD (when it shouldn’t) ?

1 Like

@Spinmoon

Welcome to the community!

Happy/sad to read about your experiences (meaning “good” to get confirmations about the issues I’ve found but sad that nothing seems to be happening about them).

Could you please keep us up-to-date about your support case with Broadcom?

In case you aren’t aware of it, Level1Techs has a YouTube channel by @wendell and a few others. I’m just an ordinary small end user without any influence but I pesterd Wendell about my absurd experiences with Broadcom HBAs and their Technical Support and he could verify something similar back in March 2023:

I hope that all this shit can be dragged out into the sunlight so that Broadcom is publicly forced to fix their darn products.

I don’t think asking for products to function as advertised is asking for the Moon.

Does your HBA’s “heartbeat LED” still flash with about 1 Hz?

sixcorners-Question-2
What settings are you supposed to use for this golden finger redriver thing? Output swing? Flat gain? Equalizer?