Generic PEX88048 cards may ship with misconfigured EEPROM (Gen1 x4 upstream)

High-Performance Computing Machine Learning, LLMs, & AI
,
1

Due to frustration, claude is responsible for this document.

The problem

I bought a “henryreal” branded PEX88048 card from Amazon (~$240). It enumerates as a Broadcom PEX880xx [1000:c012], the switch topology appears in lspci -tv, fans spin on both R9700s (power through C-Payne device adapters is fine) — but zero GPUs appear on any downstream port.

The root cause is in the upstream port’s capability register:

=== 01:00.0 (upstream port) ===
LnkCap:  Port #0, Speed 2.5GT/s, Width x4
LnkSta:  Speed 2.5GT/s, Width x4
LnkCap2: Supported Link Speeds: 2.5GT/s

This is a Gen 4 switch chip advertising Gen 1 x4 as its maximum capability. That’s not a negotiation failure — it’s the EEPROM configuration. The chip can do Gen4 x16, but the board’s EEPROM tells it not to. All downstream ports also show 2.5GT/s capability only.

For comparison, a working card should show Speed 16GT/s, Width x16 in LnkCap.

Full diagnostic

# Upstream port
=== 01:00.0 ===
LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L1 <32us
LnkSta: Speed 2.5GT/s, Width x4

# Downstream ports (all empty — no GPUs detected)
=== 02:00.0 ===
LnkCap: Port #224, Speed 2.5GT/s, Width x16, ASPM L0s L1
LnkSta: Speed 2.5GT/s, Width x16

=== 02:04.0 ===
LnkCap: Port #225, Speed 2.5GT/s, Width x16, ASPM L0s L1
LnkSta: Speed 2.5GT/s, Width x16

=== 02:0c.0 ===
LnkCap: Port #227, Speed 2.5GT/s, Width x16, ASPM L0s L1
LnkSta: Speed 2.5GT/s, Width x16

=== 02:1c.0 ===
LnkCap: Port #255, Speed 2.5GT/s, Width x16, ASPM L0s L1
LnkSta: Speed 2.5GT/s, Width x16

What I checked

  • Card is physically x16 edge connector, fully seated in CPU-direct Slot 1 (X870E Tomahawk, Gen5 capable)

  • Reseated the card, tried BIOS slot speed at Gen4 and Auto — no change

  • No DIP switches on the board to change upstream config

  • GPUs have power (fans spin from C-Payne 6-pin slot power + 12VHPWR from PSU)

  • pci=realloc in GRUB, Above 4G + ReBAR enabled in BIOS

  • Vendor website unreachable, no firmware download path

The takeaway

Not all PEX88048 carrier boards ship with correct EEPROM programming. The Broadcom chip is fine — the board vendor’s firmware is wrong. There’s no user-serviceable fix without Broadcom’s proprietary SDK tools.

If you’re buying a PEX88048/88096 for GPU use, run lspci -vv on the upstream port before building anything around it. LnkCap should show 16GT/s (Gen4) and your expected width. If it shows 2.5GT/s, return it.

Setup context

  • ASRock X870E Tomahawk MAX, Ryzen 9 9900X, 96GB DDR5

  • 2× Radeon AI PRO R9700 (gfx1201) on C-Payne SlimSAS Gen4 device adapters

  • Ubuntu 24.04.4, kernel 6.17 HWE, ROCm 7.2.3

  • Goal: dual-card DDP/FSDP training via RCCL

2

Hi Waylon, I saw your work with the PEX88048 EEPROM configuration. Have you done any work with PEX89000/PEX89144 Gen5 switches? I’m looking at repurposing Dell XE9680 PEX89144 switch boards and would appreciate any advice or issues you think I should watch for.

3

Afaik c-payne does have access to broadcomm stuff for pcie switches because he sells boards with switches on them and ive seen people sharing emails about asking him to provide firmware to enable different lane splitting for their usecase, where he could configure that.

Maybe try asking him

4

I’m not convinced the Gen1 x4 upstream is the real culprit. Even at 2.5GT/s x4, devices behind the switch should still enumerate, just slowly. Zero GPUs means the downstream links aren’t training at all.

The numbers also look off: 4× x16 downstream plus x4 upstream adds up to 68 lanes on a 48-lane chip. That suggests the switch isn’t loading any valid config and is running on defaults, rather than having a “wrong” config.

I’d check lspci -vvv -s 02:00.0 for DLActive/PresDet, and dmesg for link-down errors. If you see PresDet+ with DLActive-, the usual suspects are the SlimSAS adapters: REFCLK/PERST# passthrough, SRIS vs. common clock, or cable lane ordering.