A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

IDK how you do your stuff, but I put my full kernel and initramfs on ESP and can boot from there. I even get my NICs initialized (from OS) before mounting root partition.

Can’t deny or support your claim because I have only dissected one bitlockerd system so far, but it had a separate NTFS partition beside the bitlockerd NTFS and I’m pretty sure it can also boot into recovery from that alone so… (X) doubt.

Hello everyone .:slightly_smiling_face:
First post :cowboy_hat_face:. Thanks a lot for documenting and sharing your tests with the internerd.

A few weeks ago, I was on vacation in the US when @wendell made a video about these disks on sale, and I bought some for my computers before going back home:
Intel Optane 905P Series 1.5TB, 2.5" x 15mm, U.2, PCIe 3.0 x4, 3D XPoint Solid State Drive (SSD) SSDPE21D015TA01
They have an insane TBW Endurance Rating (Lifetime) of 27.37 PB Written.

I want to connect them to 3 computers. The disks are PCI 3 x4, and with the TBW, the cables or adapters will continue working with the disks until they die, so there is no need for PCIE 4 speed; if in the future I need PCIE4 speed, I will buy new adapters, cables.

  1. Gigabyte X370 K7 with 3950x already has one U.2 connector, which replaces the m.2 connected to the CPU if used. Connect 1 disk.


    Which cable do you recommend for PCIE 3 x4 speeds? Can one of Aliexpress work at this speed?

  2. Asrock x570 Taichi with 5950x, use the first or second m.2 to connect one disk.
    Which cable or adapter do you recommend for PCIE 3 x4 speeds? Can one of Aliexpress work at this speed?

  3. Asrock B550M Phantom Gaming with 5700G and Asrock HYPER QUAD M.2 CARD 4x4 m.2 card on the first PCIEx16 slot for 3 disks and another disk on the first M.2 slot.

I know 5000 APUs can only bifurcate to 8x4x4 the first PCIEx16 slot. I have this PC with 4 M.2 running fine, 3 on the 4x4 card, and one on the first M.2 slot running PCIE 3 x4 speed. I want to connect 4 U.2 to the same M.2 connectors.
Which cable or adapter do you recommend for PCIE 3 x4 speeds? Can one of Aliexpress work at this speed?

From a gigabyte x399 designare, I have an m.2 to U2 adapter (same U.2 connector as in the previous picture), x399 was PCIE 3.

Note: I can not post any links.

Thanks!

You are doing incredible things on this thread with things like 8 U3 gen5 SSD on a backplane on epyc servers so i’d like to share something completly different.

This is a 2x m2 nvme to pcie X1, with no bifurcation needed thanks to a pcie switch, for less than 15$ :

https://fr.aliexpress.com/item/1005006122946319.html

It’s working perfectly.

I’m using it on my backup server with an Asrock N100M motherboard so i can have more sata ports and an additional m2 SSD while keeping the X4 slot for a NIC :

( And to be honest i was just curious to try it. )

I’m always amazed by the weird things you can find on aliexpress…

2 Likes

Welcome to the forum!

I hope I’m not messing up, am a little under the weather with brain fog.

I have basically practically tested everything you asked about myself, have to go through my notes

Short answer: To avoid waste even for PCIe Gen3 it’s not advisable to just buy from “unkown suppliers”, but PCIe Gen3 is possible with passive cables and adapters without any active ReTimers/ReDrivers.

  1. Regarding a cable for an U.2 SFF8643 port to SFF8639 NVme SSD - have to look a little, there have been sooo many disappointments.

  2. Connecting U.2 SSDs to M.2 slots, these adapters work fine for me up to Gen3, they DO NOT actually handle PCIe Gen4 without PCIe Bus Errors:

PCIe Gen4 U.2 or U.3 SSDs work with these adapter cables at Gen4 speeds but get errors under load. Strangely the one with the 50 cm cable produces less errors for me compared to the 20 cm model, but I only have one unit of each variant.

1 Like

Nice little unit, although it does seem to be only PCIE 2.0.

Also a handy little product! Do you have a link for that M.2 to SATAx6 adaptor?

I’m not the other user but all M.2 6-port SATA HBAs I’ve seen so far are a “meh” design with an ASMedia HBA chip and a SATA port multiplier.

I’m getting another M.2 variant for a small NAS soon where no PCIe slot is left and will likely go with a JMicron variant with 5 native ports and a PCIe Gen3 x2 interface.

https://www.delock.de/produkt/64051/merkmale.html?setLanguage=en

Already have a little PCIe AIC with same JMicron chip for testing and it is working fine.

1 Like

Yes, this card is PCIE 2.0, confirmed with lspci, this is viable only when speed is not the primary concern :wink:

The M2 to SATA is this one, based on the ASM11666, it has a PCIE 3.0X2 interface so you will be speed limited if you plug 5 SSD but i have used these in 3 different systems, with no problem to report :

https://fr.aliexpress.com/item/1005005427076687.html?gatewayAdapt=glo2fra

https://www.asmedia.com.tw/product/45aYq54sP8Qh7WH8/58dYQ8bxZ4UR9wG5

1 Like

Checked out an M.2 6 x SATA ASM1166 adapter from Amazon, is indeed faster than the “old” JMB 5 x SATA thingy (564 MB/s read instead of 544 MB/s with the same SSD).

Could be updated without any issue to the latest known public firmware 20221118 which is a few versions newer than the one SilverStone offers on their website.

1 Like

Good stuff.
I suspect you’ve already tested parallel disk I/O? With a pair of gen 3 lanes, it would in theory be able to max out 3 SATA devices, and almost max out a 4th, but maybe that’s not how it works in practice…

1 Like

I will try to free up 6 performant SATA SSDs to test it, don’t have 12 at hand.

Made a PCIe x8 Bifurcation-to-2xM.2 Ghetto 12 x SATA HBA with two ASM1166 M.2-6xSATA adapters.

Will never get anything again from Broadcom if I can avoid it (i. e. not needing SAS features or just operating slow mechanical HDDs):

9 Likes

That should be in the gore section that’s awesome.

4 Likes

Just wanted to see if you managed ever to get the 1200p for testing.

I wouldn’t even consider it gore. My reasoning:

  • The Bifurcation HBA is used in a slot with CPU PCIe where PCIe Advanced Error Reporting is validated to work.

  • The used PCIe x8-to-2xM.2-Bifurcation adapter (Delock 89837) is capable of doing PCIe Gen4 without causing PCIe Bus Errors

  • The used ASM1166 M.2 SATA adapters only use PCIe Gen3 x2.

  • The ASM1166 is widely spread globally so firmware fixes should be accessable for a long time.

  • Contrary to Broadcom HBAs this DIY ASM1166 thingy hasn’t caused a single system crash in my testing, yet.

The only thing I don’t like:

  • One ASM1166 chip reports having 32 SATA endpoints (not just the 6 that actually physically exist) to the operating system (maybe for the use with separate port multipliers?) so you should practically test if the OS of your choice doesn’t bug out here.

Digi-Key US still blocks my orders, I’m trying to get it solved with the European Digi-Key support. Orders for the 1200p-32i get automatically transferred to Digi-Key US which for some reason doesn’t like me, don’t even know why.

Does microchipdirect ship to Germany? I’m about to order the x8 U.2 cable from them because digikey is out of stock of it.

1 Like

It will be my last resort (oh God, now I’m remembering the Papa Roach song) if the current thing with DigiKey Europe also falls through. Looked through the Microchipdirect ordering process, at first the part is available to ship to Germany and when you get to the last step the estimated delivery date suddenly jumps up x weeks.

Of course, had I known the DigiKey shenanigans back in January all shipping and customs processing would be done by now.

I still want to get the DigiKey stuff in order since I can see myself ordering other stuff from them in the future.

Yay, I got a shipping notification and UPS tracking information, the bureaucracy seems to have been resolved, finally.

3 Likes

The 1200p-32i arrived yesterday, imported from Kentucky, DigiKey doesn’t skimp on the shipping (two days between shipping notification and delivery).

Still getting to know my way around the Adaptec mannerisms, what sucks is that Adaptec’s support explicitly lied to me:

  • Drives get abstracted even in “Raw” mode, similar to Broadcom HBAs 9400 and newer
  • CrystalDiskInfo and SSD manufacturer tools for firmware updates etc. don’t detect SSDs connected via the HBA

But Adaptec’s management software is actually useful, Broadcom’s LSA is a piece of trash.

2 Likes

This burns…

:people_hugging:

2 Likes

Copied question & answer from my interaction with Adaptec’s support back in January 2024:

Q: Does the HBA Ultra 1200p-32i act completely transparently regarding connected SSDs?

Broadcom Tri-Mode HBAs abstract connected SSDs to kind of generic block devices so SSD manufacturer firmware update or service tools cannot detect connected SSDs. A similar situation exists regarding standard SMART monitoring tools like CrystalDiskInfo under Windows 10 22H2/Windows 11 23H2.

A: Yes, the service tool should be able to see all SSDs and you would be able to manage them through their proprietary tool.

:frowning:

BUT so far at least not a single crash or issue in a little over a day of testing. No comparison to Broadcom…

Can’t believe that after a few years I can finally complete a build with an AM4 5950X system that is supposed to handle 8 PCIe Gen4 U.2 SSDs in hot-swappable bays without having to reboot the system.

:wink:

1 Like

CrystalDiskInfo doesn’t pick up any of the info from the PCIe connected drives?
The Ultra 3258p-32i does, I would have expected the HBA card to function similarly to the RAID card just without the RAID part.