i fell in the rabbit hole 
I’ve been working on getting MI250/MI250X OAM modules running on standalone PCIe carriers.
After reverse-engineering both the module SPI image and the Supermicro H12DGQ-NT6 BMC firmware, we now have a much clearer picture of the module’s boot path and what the host/platform is expected to provide.
Anything marked [VERIFIED] below was checked byte-level against the module SPI dump and/or BMC image.
Anything marked [CURRENT HYPOTHESIS] is an interpretation that still needs hardware confirmation.
TL;DR
1. The MI250X PSP self-boots from the module SPI [VERIFIED]
The module contains the initial secure boot chain locally:
PSPBL_STG1
↓
ARM bootstrap stub
↓
VBL1 / VBL2 / VBL3
↓
MP1 / SMU
The VBL blobs contain the HBM/UMC-related initialization logic identified in the dump, along with anti-rollback/security material.
The important distinction is:
No host-side firmware transfer appears to be required to start the initial PSP/VBL boot chain.
This does not mean the Linux driver never transfers firmware or talks to the PSP later. AMDGPU has later PSP/runtime-firmware handshakes; those are separate from the module’s initial self-boot.
This retires the earlier theory that the host must upload firmware just to get the PSP started.
2. Three host-side prerequisites are directly demonstrated by the current boot path [VERIFIED]
So far, the firmware path directly shows that the host must provide at least:
-
Correct power / clock / reset sequencing.
-
A usable MMIO path and execution of ATOM ASIC_Init.
-
A valid RCC_CONFIG_MEMSIZE.
There may still be additional platform prerequisites, particularly on the sideband-management side. Those are not yet proven necessary for POST.
ASIC_Init does not perform HBM training.
Instead, it waits for the PSP state, derives framebuffer aperture state from RCC_CONFIG_MEMSIZE, enables BIF_FB_EN, and sets the ASIC-init-complete scratch bit.
3. The H12DGQ-NT6 BMC gives us the observed sideband management protocol [VERIFIED]
The BMC image exposes the relevant GPU-management activity on SMBus/I2C bus 2:
-
mux topology
-
module-controller probing
-
baseboard FRU
-
FPGA SMBPBI proxy
-
PLDM/MCTP firmware-update path
-
CEC update controller
-
power/thermal management interaction
-
retry and timing constants
This strongly suggests that the necessary subset could be reproduced by a carrier MCU/CPLD.
Which subset is actually required for POST is still an open question.
4. Current leading hypothesis for the 20-second ATOM hang
A typical failure looks like:
atombios stuck executing 2488
followed by -22.
Given the self-booting PSP/VBL chain, the leading hypothesis is:
The PSP never reaches the state that ASIC_Init is waiting for.
The likely causes are now much more hardware/platform-oriented:
-
incomplete power sequencing
-
missing or invalid power-good signaling
-
clocking
-
reset timing
-
MMIO access to the relevant register path
-
possibly another platform prerequisite we have not identified yet
There is a simple register test that should divide the problem space very quickly.
1. Module Side: PSP Self-Boot
1.1 PSP directory [VERIFIED]
PSP directory location in the HPE MI250X dump:
ROM offset 0x079C
Format:
offset size field
0x079C 4 "$PSP"
0x07A0 4 checksum = 0x4985C6AD
0x07A4 4 count = 0x24 (36)
0x07A8 16*36 entries:
u32 reserved
u32 id
u32 size
u32 offset
The size/offset pairs form a gapless blob chain, and the individual blobs contain the expected signatures.
Selected entries:
id 0x01 PSPBL_STG1 @0x2D000 size 0x8000
id 0x03 second "$PS1" BL @0x88000 size 0x8000
id 0x04 legacy VGA/ATOM area @0x00000 size 0x2800
id 0x23 ARM bootstrap stub @0x90000 size 0x8000
id 0x2E VBL run config @0x47000 size 0x10000
id 0x40 VBL_STG1 @0x99000 size 0x4000
id 0x41 VBL_STG2 @0x9D000 size 0x3000
id 0x42 VBL_STG3 @0xA0000 size 0x3000
id 0x46 MP1/SMU firmware @0xA3000 size 0x20000
id 0x55 ANTI_ROLLBACK @0xA0000 size 0x3000
Interesting details:
PSPBL_STG1
↓
ARM stub 0x23
↓
VBL1
↓
VBL2
↓
VBL3
The ARM bootstrap clears the boot mailbox at:
0x078041CC
then writes:
0x1234ABCD
before entering the VBL flow.
HBM/UMC-related logic is present inside the VBL blobs rather than in a separate ATOM memory-training table.
The retail MI250 image D6521000V.073 has the same 36-entry directory, IDs and offsets.
Of the 36 blobs:
6 are byte-identical
30 differ
Importantly:
ASIC_Init is byte-identical between the retail MI250 and HPE MI250X images.
1.2 ATOM Tables [VERIFIED]
HPE:
ATOM_ROM_HEADER v2_2 @ 0x4AC
Retail:
ATOM_ROM_HEADER v2_2 @ 0x4B2
Master command table:
0x23E2
Of the 81 possible command-table slots, only five are populated:
idx 0 ASIC_Init @0x2488
idx 6
idx 22 SMN/clock helper
idx 41 stub
idx 53 MP1 mailbox accessor
Among the populated ATOM tables, ASIC_Init is the one directly involved in the observed POST path.
There is no ATOM-level HBM training table.
The master data table contains 14 entries including:
firmwareinfo
smc_dpm_info
smu_info
gfx_info
umc_info
vram_info
voltageobject_info
powerplayinfo
...
1.3 ASIC_Init @ 0x2488 [VERIFIED]
Decoded table:
40 00 02 01 00 08
size 0x40, frev 2, crev 1, WS=0, PS=8
37 00 00
setport MMIO
4B E5 00 04
TEST PS[0] & 0x04
(parameter byte 24; corresponds to PS[0] bit 26)
49 20 00
JMP NOTEQUAL -> skip PSP wait
01 05 00 00 61 60 01 00
MM_INDEX = 0x00016061
3C E5 01 00 80
compare upper byte of MM_DATA against 0x80
49 18 00
loop until equal
03 00 41 E3 0D
WS[0x41] = RCC_CONFIG_MEMSIZE
33 05 41 01 00 00 00
WS[0x41] -= 1
1B 05 41 0B
WS[0x41] >>= 11
01 02 06 00 41
MM_INDEX_HI = WS[0x41]
0D 25 20 0E 03
BIF_FB_EN |= 0x03
0D 65 53 00 02
BIOS_SCRATCH_7 |= 0x200
ASIC_INIT_COMPLETE
5B
EOT
The register involved is:
MP0_SMN_C2PMSG_33
AMD’s register index is:
0x16061
which corresponds to MMIO byte offset:
0x58184
The interesting detail is that the ATOM table compares the entire upper byte against:
0x80
rather than merely testing bit 31.
So the actual ATOM wait condition is effectively:
(value >> 24) == 0x80
Diagnostic Escape Path [VERIFIED]
There is a very useful branch at the start of ASIC_Init.
If the caller sets:
PS[0] bit 26
or:
0x04000000
the table skips the MM_INDEX setup and PSP polling loop.
It still executes:
RCC_CONFIG_MEMSIZE handling
MM_INDEX_HI
BIF_FB_EN
BIOS_SCRATCH_7 / ASIC_INIT_COMPLETE
This behaves like an already-posted / skip-wait path.
That makes it very useful diagnostically.
It does not magically make a dead PSP work.
What it does is move initialization past the ATOM polling loop so that the later AMDGPU PSP/runtime-firmware path can tell us where things fail next.
2. H12DGQ-NT6 BMC Sideband Behavior [VERIFIED]
All observed GPU-management activity is on SMBus/I2C bus 2.
Relevant addresses seen in the BMC code:
role 7-bit address
sensor mux PCA9548 0x71
GPU slot mux PCA9548 0x72
PCA9546 muxes 0x75 / 0x77
AOM baseboard CPLD 0x54
module management controllers 0x4C-0x4F
0x44-0x47
baseboard FRU 0x50 / 0x51
fan / thermal controller 0x29
power / voltage / current monitor 0x28
FPGA SMBPBI proxy 0x60
PLDM / MCTP endpoint 0x58
CEC update controller 0x55
GPU-slot mux behavior at 0x72:
ch0 = slots 1-2
ch2 = slots 3-4
ch4 = slots 5-6
ch6 = slots 7-8
open = write 1 << channel
Baseboard CPLD at 0x54:
mux register = 0x40
open:
0x40 | n
close all:
0xFF
The BMC probes the module-controller addresses with command 0x00 plus a read.
Important distinction:
Those are module management controllers being probed by the BMC. A replacement carrier should first try to expose/route the actual module-side devices, not blindly emulate them.
SMBPBI Proxy [VERIFIED]
BMC → FPGA at 0x60:
1. Write 6 bytes:
5C 04 <bytecount> <command> <arg> 80
2. Write:
5C
then read 5-byte status
3. Write:
5D
then read 5-byte data
Observed status handling includes:
(b & 0x1f) == 0x1f
for not-ready/retry handling.
Statuses observed in the image include:
0x1E30
0x1E78
0x25B6
with 0x1E handled as busy.
PLDM / Firmware Update [VERIFIED]
The BMC contains a normal firmware-update flow:
GetFirmwareParameters
↓
RequestUpdate
↓
PassComponentTable
↓
UpdateComponent
↓
TransferComplete
↓
VerifyComplete
↓
Activate
Endpoint:
0x58
This is part of the server’s firmware-management path and should not be confused with the PSP’s initial self-boot.
CEC Update Controller [VERIFIED]
Controller:
0x55
The BMC:
-
probes its version,
-
retries up to three times with ~1 s delay,
-
sends either opcode:
0x80
or:
0x06
with payload:
0x88 / 0x0E
using SMBus PEC.
BMC State / Timing Model [VERIFIED]
Observed state model:
0 = absent
1 = powered
2 = firmware update in progress
3 = idle
The BMC does not appear to directly bit-bang every power rail.
The baseboard FPGA/CPLD performs the lower-level sequencing, while the BMC drives the higher-level state/control logic and performs:
mux control
device probing
SMBPBI
PLDM
CEC
monitoring
Observed timing constants include approximately:
100 ms discovery retries
10-30 ms SMBPBI command retries/phases
2 ms status polling in one path
30 ms between command phases
3. Why the Standalone Carrier Is Probably Hanging
[CURRENT HYPOTHESIS]
The 20-second:
atombios stuck executing 2488
timeout means ASIC_Init never sees its expected C2PMSG_33 value.
The current possibilities are:
1. PSP never reaches ready state
Leading candidate.
Potential reasons:
power sequencing
clocking
HOST_PWRGD
MODULE_PWRGD
PVREF
PERST# timing
other OAM platform state
The expected OAM sequence we’ve been working from is approximately:
P48V
↓
P3V3
↓
HOST_PWRGD
↓
module rails
↓
PVREF
↓
MODULE_PWRGD
↓
clocks
↓
PERST#
with PERST# held until after the appropriate power-good/clock conditions.
2. MMIO/register access problem
The GPU may be alive, but the host’s access to:
MP0_SMN_C2PMSG_33
may not be producing the value ATOM expects.
The fact that PCI enumeration and ROM-BAR access work makes a total PCIe failure unlikely, but it does not prove this register path is correct.
3. RCC_CONFIG_MEMSIZE is invalid
This would break the framebuffer-aperture setup after the wait.
It does not explain the polling loop itself, because RCC_CONFIG_MEMSIZE is consumed after the PSP wait exits.
4. Another platform prerequisite exists
The BMC work shows substantial active sideband management on a real server.
We do not yet know whether some part of that environment is a hard prerequisite for POST.
4. Tests to Run
TEST 0 — Read the PSP state register
This is the highest-value test.
Read GPU MMIO byte offset:
0x58184
which corresponds to AMD register index:
0x16061
and:
MP0_SMN_C2PMSG_33
Ideally do this through a tiny kernel module/debug driver, or another method that safely maps the GPU MMIO BAR before normal AMDGPU initialization.
Interpretation:
Upper byte == 0x80
This is the exact condition the ATOM table is waiting for.
If this value is present while ATOM is still spinning, the problem is probably the access path or ATOM execution rather than PSP startup itself.
Upper byte == 0x00
The device has not reached the value ATOM expects.
That is consistent with the PSP not starting or not completing early initialization, but by itself it does not prove the power sequence is the cause.
Any other value
Post the complete 32-bit value.
Do not assume 0x81+ automatically means “PSP error.”
The current Linux code only tests bit 31 for its own early wait, while this ATOM table specifically compares the upper byte against 0x80.
Other values therefore deserve to be decoded rather than pre-classified.
Also capture:
0x14C BIOS_SCRATCH_7 / ASIC_INIT_COMPLETE bit 9
0x5A090 TOPOLOGY_ID
plus complete dmesg.
TEST 1 — Power/timing experiment
As a diagnostic experiment:
1. Power the module's 48 V supply first.
2. Wait 30-60 seconds.
3. Boot the host.
4. Repeat TEST 0.
This is not proposed as the proper final OAM power sequence.
It is simply a useful test.
If the PSP-state register changes when the module receives power substantially earlier, that strongly implicates sequencing/timing/clock availability.
TEST 2 — Use the ASIC_Init escape branch
Before atom_execute_table() executes ASIC_Init, set:
PS[0] |= 0x04000000
Alternatively patch the corresponding VBIOS TEST condition.
This bypasses the PSP-ready polling section while still allowing the remainder of the table to execute.
Then record exactly where AMDGPU fails next.
The purpose is diagnosis, not to claim the bypass itself is a valid initialization sequence.
TEST 3 — Explicitly enable MC/BIF access
For Aldebaran, experiment with:
nbif_v6_3_1_mc_access_enable(adev, true)
instead of relying exclusively on the normal ATOM path.
The function contains the BIF_FB_EN programming we care about.
This is more invasive than TEST 2 but may help separate:
PSP wait
from:
framebuffer/BIF aperture enable
behavior.
TEST 4 — Sideband Visibility
For carrier owners with access to the relevant SMBus:
Check whether the expected paths/devices are visible.
Relevant addresses:
0x50 / 0x51 baseboard FRU
0x54 baseboard CPLD
0x4C-0x4F
0x44-0x47 module-management devices
0x58 PLDM/MCTP path
0x60 SMBPBI proxy
0x55 CEC controller
The goal is to determine what actually exists on the custom carrier/module combination, not simply to fake every address.
In particular, if the module-management controllers are physically present on the OAM, the carrier needs to expose the correct bus path to them.
If the carrier lacks the expected baseboard FRU, we can separately test presenting the recovered AOM-MCM-Q FRU image using an EEPROM such as an AT24C04 at 0x50.
TEST 5 — Retail MI250 vs HPE MI250X A/B
Test on the exact same host/carrier:
Retail MI250 0x740C
HPE MI250X 0x7408
The ASIC_Init tables are byte-identical.
A same-rig A/B test therefore gives us an unusually clean way to separate:
module/platform behavior
from:
ATOM table behavior
The retail MI250 has reportedly enumerated farther on a custom carrier, so reproducing that comparison under controlled conditions would be extremely valuable.
TEST 6 — Firmware Hygiene
This is not expected to fix the ATOM wait, because the relevant ASIC_Init table is byte-identical.
It is still useful for removing firmware-version variables.
Important distinction:
The current AMD FW Flash documentation lists:
MI210 IFWI / RMFW
MI250 IFWI / RMFW
MI250X RMFW only
So for an MI250X, do not assume:
amdfwflash --update-ifwi ...
is a supported route.
The Shoreline RMFW image identified in the package is:
113-SHOD652010Cxx.010.update.bin
which is byte-identical to:
AMDD652050Cxx.007
and contains the 3.3 V brown-out-detector fix documented by AMD.
If MI250X IFWI itself needs to be changed on a standalone HPE module, that requires a different update path — e.g. the original platform/vendor mechanism if available, or direct SPI programming for offline experimentation.
Again:
firmware hygiene is not currently the leading explanation for the POST failure.
5. What a Standalone Carrier Actually Needs
This part is deliberately split by confidence level.
A. Directly indicated by the current boot investigation
A working carrier needs to provide correct:
48 V / 3.3 V power
power sequencing
power-good signaling
required clocks
reset timing
PCIe connectivity
MMIO access
including the OAM power/reset relationships around:
HOST_PWRGD
MODULE_PWRGD
PVREF
PERST#
AUX reference clock
This is currently the first thing I would reproduce correctly.
B. Baseboard identity / management functions likely worth implementing
The original platform provides:
FRU @ 0x50 / 0x51
PCA9548 @ 0x72
CPLD @ 0x54
A small MCU/CPLD can likely reproduce these roles for a standalone carrier.
However, whether all of them are required before POST is not yet proven.
C. Module-management devices
The original BMC probes devices at:
0x4C-0x4F
0x44-0x47
The first goal on a custom carrier should be:
make the actual module-side management devices reachable and probe them.
Do not emulate them unless hardware testing demonstrates that emulation is actually necessary.
D. Full server-management emulation
For a more complete server-platform illusion, we would eventually want:
SMBPBI proxy @ 0x60
PLDM/MCTP @ 0x58
CEC @ 0x55
These are clearly part of the production management environment.
They are not yet proven to be required for GPU POST alone.
6. Artifacts / Hashes
HPE MI250X SPI dump:
hpe-vbios.bin
SHA-256:
d8132a848d245d2679c9a1a755f3cbda959489a41a373dfd6c27c8aa874e81b7
size:
2 MB
config:
D6520100-100
Retail MI250:
D6521000V.073
internal config:
D65210V-073
SHA-256:
a57afa712d6869277e76f991dcc2e7b19faa6f47573f6857c3727bfc49fc09a4
This image is byte-identical to the corresponding public production amdfwflash IFWI image.
MI250X-related IFWI image under investigation:
D6520100.050
SHA-256:
5208b46d0ad2d9662034eaabfbdc65eb8b97a1e85f048119f71eb38f79634ae4
Shoreline RMFW:
113-SHOD652010Cxx.010.update.bin
SHA-256:
44da697162a1a85982187929ddaaf3165dab1ef4e9c3f8cd4157fed36c0d3608
Byte-identical to:
AMD AMDD652050Cxx.007
Baseboard FRU:
AOM-MCM-Q
512 bytes
I2C 0x50
Reference server firmware:
Supermicro H12DGQ-NT6
BIOS 3.6
BMC 01.06.33
Full module/BMC reverse-engineering reports are available separately.
7. Open Questions / Help Wanted
1. Does MODULE_PWRGD ever assert correctly on the cheap carriers?
A logic-analyzer capture of:
HOST_PWRGD
MODULE_PWRGD
PERST#
relevant clocks
during startup would be extremely useful.
2. PCIe AtomicOps
Does the lack of the required PCIe AtomicOp capability/routing on the consumer host only matter later in initialization, or is it a hard blocker earlier than expected?
Testing on a known-good dual-EPYC platform should answer this.
3. Is the sideband management environment required for POST?
Specifically:
FRU
CPLD
SMBPBI
PLDM
CEC
Which of these, if any, must exist before the GPU reaches normal AMDGPU initialization?
4. EX235a FRU
If anyone has access to an HPE EX235a, please dump the original baseboard FRU.
If the module validates FRU identity/content, comparing it against the AOM-MCM-Q image could be important.
5. HPE EX235a BIOS 2.3.0
I’m looking for the 16 MB node BIOS so we can inspect the PEI modules and their register programming.
Particularly interesting symbols/modules include:
PspBarInitEarlyV2
AmdPspApobLibConstructor
FabricSetMmioReg3
That would give us another reference for the host-side early-init sequence.
8. Bottom Line
The strongest result so far is:
The MI250/MI250X module contains its own initial PSP/VBL boot chain and does not appear to require the host to upload firmware merely to start the PSP.
The observed startup model is now:
power / clocks / reset
↓
module PSP/VBL self-boot
↓
HBM/UMC-related initialization
↓
C2PMSG_33 reaches expected state
↓
ATOM ASIC_Init exits its wait
↓
RCC_CONFIG_MEMSIZE
↓
BIF_FB_EN / framebuffer aperture
↓
ASIC_INIT_COMPLETE
↓
AMDGPU continues
↓
later PSP/runtime firmware interactions
A production server additionally provides an active sideband-management environment through its BMC/baseboard logic.
What we do not yet know is exactly how much of that management plane is mandatory for POST.
So the current leading theory for the 20-second ASIC_Init @ 0x2488 hang is:
the module never reaches the PSP state that ATOM expects, most likely because the standalone carrier is missing part of the required electrical/platform bring-up sequence.
But that’s still a hypothesis.
The fastest way to move this forward is to read:
MP0_SMN_C2PMSG_33
register index: 0x16061
MMIO byte offset: 0x58184
and post the full 32-bit value, along with:
BIOS_SCRATCH_7
TOPOLOGY_ID
dmesg
carrier model
host platform
That should tell us whether to spend the next round of work below the driver — power/clocks/reset/platform — or above it in ATOM/AMDGPU.
Let’s get these things booting.