A Neverending Story: PCIe 3.0/4.0/5.0 Bifurcation, Adapters, Switches, HBAs, Cables, NVMe Backplanes, Risers & Extensions - The Good, the Bad & the Ugly

I realised I do have one AER-capable PCIe 4.0 x16 slot - in my ASRock B650M-HDV/M.2 motherboard of all places! The MB is so sparse on peripherals that four of the CPU lanes are actually routed to slot 5!


(Image stolen from The Overclock Page.)

So I got that Kalea 2x MCIO card (via Ebay) and also a Supermicro CBL-MCIO-1240U2Y-E cable to try out:


The card is really a “Shinreal PCIe x16 to 2x MCIO 8i Adapter”, “XIC PE5160-4IL”:

I made sure AER was enabled and PFEH disabled in UEFI:


… inserted the card and hooked my PCIe Gen3 P4610 U.2 drive up. I had to disable MB Thunderbolt support which apparently is via some optional add-on card meant to go into PCIE3 (where the card was). A thing that threw me off at first was that the Supermicro cable has the wrong markings - the first port is the one marked Port1 and the second is the one marked Port0! :facepalm: Disappointing; you should do better, Supermicro!

Anyway, I booted up Fedora, installed kdiskmark and ran a benchmark on the drive while running ‘journalctl -f’ to catch any errors. Everything went fine!

The drive also linked up at Gen3 speeds:

# lspci -vv -s 04:00.0
04:00.0 Non-Volatile memory controller: Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller] (prog-if 02 [NVM Express])
	Subsystem: Oracle/SUN NVMe PCIe 3.0 SSD v2 6.4TB 2.5-inch (P4610)
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 84
	IOMMU group: 16
	Region 0: Memory at f6f10000 (64-bit, non-prefetchable) [size=16K]
	Expansion ROM at f6f00000 [disabled] [size=64K]
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [50] MSI-X: Enable+ Count=129 Masked-
		Vector table: BAR=0 offset=00002000
		PBA: BAR=0 offset=00003000
	Capabilities: [60] Express (v2) Endpoint, IntMsgNum 0
		DevCap:	MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 75W TEE-IO-
		DevCtl:	CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+ FLReset-
			MaxPayload 512 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr+ NonFatalErr- FatalErr- UnsupReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x4, ASPM L0s, Exit Latency L0s <64ns
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x4
			TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-

Also AER seemed to be working:

# lspci -tv
-[0000:00]-+-00.0  Advanced Micro Devices, Inc. [AMD] Device 14d8
           +-01.2-[04]----00.0  Intel Corporation NVMe Datacenter SSD [3DNAND, Beta Rock Controller]
# dmesg | grep AER
[    0.269761] acpi PNP0A08:00: _OSC: OS now controls [PCIeHotplug SHPCHotplug PME AER PCIeCapability LTR DPC]
[    2.335250] pcieport 0000:00:01.1: AER: enabled with IRQ 29
[    2.335408] pcieport 0000:00:01.2: AER: enabled with IRQ 30
[    2.335550] pcieport 0000:00:02.1: AER: enabled with IRQ 31

Conclusion: the Kalea-Informatique MCIO adapter + Supermicro CBL-MCIO-1240U2Y-E combo seems to be working fine at Gen3 speeds - even with the very long routing of the lanes on my motherboard!

(Unfortunately I don’t have access to any Gen4 drives, so I couldn’t test that. :frowning: )

2 Likes