Return to Level1Techs.com

Draft: PCIe 3.0/4.0 Bifurcation, Extensions & Adapters - The Good, the Bad & the Ugly

Hi,

I’m creating this draft of a topic to seek out feedback from other users.

If you have a question, please start your response with “Nickname-Question-#”, an example would be “aBav. Normie-Pleb-Question-1”.

This makes it easier to search “-Question-” in the thread to hopefully not overlook anyone.

My motivation for this thread is my experience with many, many, MANY PCIe adapters and headaches and now at least I think I’m beginning to recognize “patterns” to further push for bug fixes.

(And yes, sometimes it really was crazy/absurd, devoid of any human-perceivable logic…)

Note: I don’t have much experience with multi GPU configurations, I like stuffing as many regular PCIe AICs/NVMe drives into a single system as possible, using up any PCIe lane a motherboard has to offer.

Regards,
aBavarian Normie-Pleb

2 Likes

old thread is very good for info

Maybe, then I’ll change this thread to document the various concrete issues I’ve personally encountered.

Also I’m specifically looking at “consumer-grade” stuff, i. e. AM4/X570/B550 Zen 2/Zen 3 configurations.

1 Like

I’ll throw this here for reference

Motherboard: Tyan Tomcat HX S8030 (S8030GM2NE)

The following are known good items for PCIe 3.0. I don’t currently have PCIe 4.0 drives, as the price/perf hasn’t been compelling enough.

  • ASUS Hyper M.2 X16 PCIe 4.0 X4 Expansion Card Note Needs 4x4x4x4x bifurcation set in BIOS, also note that the “1st” slot referenced in BIOS is actually the one farthest away from the CPU, with the 5th one being the closest.

This tyan board has some odd 8i slimline PCIe ports. The following seems to work well though it’s currently fairly pricey:

1 Like

While its build quality is certainly nice, have experienced some issues with the ASUS Hyper M.2 x16 Gen4 AIC, too.

“Strange” is the best way to summerize the experiences I’ve had the pleasure of witnessing within last 18 months with this topic :wink:

Also got Delock Gen4 Slimline stuff around, basically anything a “normal” customer would buy hoping “That would be nice if it works as advertized” and then has the joy of being left out in the rain by less than competent customer support where the go-to answer is “we don’t support the use of our product with a different product”.

1 Like

Life has been running a bit of interference but was able to look into a thingy completely new to me:

Delock 90504 PCIe Gen3 x16 AIC with Broadcom PEX8749 PCIe switch chipset to connect up to eight x4 NVMe SSDs

  • Individual Gen3 NVMe SSDs perform close to their native speed, but the used U.2 cables influence the performance numbers. This is a bit tricky since the AIC doesn’t have any management software, drivers or anything to check for PCIe transmission errors (Nothing in Windows Event Log, PCIe NVMe SSDs that are connected with cables not up to the task can sometimes produce WHEA 17 errors).

  • Nice: Being able to properly hot-plug NVMe SSDs in Windows without the system crashing (drives can be ejected like USB thumb drives, for example)

  • Weird: The AIC doesn’t have any drivers and connected NVMe SSDs show up in Windows as they should (also in tools like CrystalDiskInfo) but it also causes four “Base system devices” Windows doesn’t find the drivers for (Code 28) to show up in Windows’ Device Manager which looks a bit improper

Does anyone have an idea what to do about that?
(@wendell ?)

The final destination for the AIC after more testing is a DIY AM4 NAS where it’s going to get eight CPU PCIe lanes for eight NVMe SSDs.

1 Like

PLX chips are extremely picky about what firmware is loaded onto them. This adapter would work better for GPUs and NICs, but not really for NVMe. The chip is meant to be fully integrated onto a motherboard like the ASUS Sage boards. If you aren’t a big vendor like ASUS, you don’t really have that expertise to tweak the PLX chip.

Linus (LTT) did a thing on One Stop Systems who do this for a living:

Since the Delock 90504 seems to be working fine and those four missing-drivers entries in Device Manager seem to be just “cosmetic” I’m pretty sure that I’m keeping it - wanna look into testing it with TrueNAS and ESXi next.

The only other thingy that would spark my curiosity would be the PCIe Gen4 Broadcom P411W-32P but using Gen4 with DIY cables and backplanes is more of a PIA and the currently intended use case doesn’t need higher sequential speeds that would require Gen4.

Strange:

  • Seems that when the Delock 90504 is connected with 8 PCIe lanes you can only connect four x4 NVMe SSDs (every other port)

  • Only when it gets 16 lanes you can use all eight ports for x4 NVMe SSDs :frowning:

Don’t know yet if I got a lemon or this is expected behavior. Had previously thought that since it has an active PCIe switch without PCIe bifurcation it doesn’t matter how many lanes it itself gets to handle attached SSDs (other than peak performance is reduced, of course).

2 Likes

Good news everyone, both issues…

  • Only up to four x4 NVMe SSDs show up when the 90504 gets an x8 instead of x16 PCIe link

and

  • The four “phantom” 87D0 PCIe devices in Windows Device Manager where drivers could not be installed (Code 28)

Have been fixed by a Broadcom firmware update you get when you annoy Delock’s Level 2 support a bit.

Hope that they’ll leave this link active:
http://www.delock.de/download/Firmware%20Update90504.rar

1 Like