Someone needs to figure out how to adapt MI250 GPUs to PCIe

My progress on this today:

Pushing an agent to modify the amdgpu driver to see how far I can get, and maybe learn a little. Anything with CRASH PROBE: is leftover from a hang/crash I was seeing before.

I managed to coax the driver into reading the vbios from the rom bar. And validated that it matches the vbios I dumped with a clip.

Now I want to caveat that while this looks like progress it sure feels like I’m drunkenly fumbling at a knob and don’t have the key. Looks like a door with a knob alright, I just can’t see the lock.

| [ 4347.614844] calling  amdgpu_init+0x0/0xff0 [amdgpu] @ 5836                                                    │
│ [ 4347.615118] amdgpu: === CRASH PROBE: amdgpu_init MODULE ENTRY ===                                             │
│ [ 4347.615257] amdgpu: Virtual CRAT table created for CPU                                                        │
│ [ 4347.615273] amdgpu: Topology: Add CPU node                                                                    │
│ [ 4347.615470] amdgpu 0000:03:00.0: amdgpu: initializing kernel modesetting (ALDEBARAN 0x1002:0x7408             │
│ 0x1002:0x0B0C 0x00…                                                                                              │
│ [ 4347.615486] amdgpu 0000:03:00.0: amdgpu: register mmio base: 0xFC800000                                       │
│ [ 4347.615489] amdgpu 0000:03:00.0: amdgpu: register mmio size: 524288                                           │
│ [ 4347.615571] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: amdgpu_discovery_set_ip_blocks ===                  │
│ [ 4347.615574] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: Aldebaran init ===                                  │
│ [ 4347.615652] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: Aldebaran IP versions set ===                       │
│ [ 4347.615656] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after amdgpu_discovery_init_soc_config ===          │
│ [ 4347.616024] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after amdgpu_discovery_sysfs_init ===               │
│ [ 4347.616028] amdgpu 0000:03:00.0: amdgpu: detected ip block number 0 <common_v2_0_0> (soc15_common)            │
│ [ 4347.616033] amdgpu 0000:03:00.0: amdgpu: detected ip block number 1 <gmc_v9_0_0> (gmc_v9_0)                   │
│ [ 4347.616037] amdgpu 0000:03:00.0: amdgpu: detected ip block number 2 <ih_v4_2_0> (vega20_ih)                   │
│ [ 4347.616040] amdgpu 0000:03:00.0: amdgpu: detected ip block number 3 <psp_v13_0_0> (psp)                       │
│ [ 4347.616044] amdgpu 0000:03:00.0: amdgpu: detected ip block number 4 <smu_v13_0_0> (smu)                       │
│ [ 4347.616048] amdgpu 0000:03:00.0: amdgpu: detected ip block number 5 <gfx_v9_0_0> (gfx_v9_0)                   │
│ [ 4347.616051] amdgpu 0000:03:00.0: amdgpu: detected ip block number 6 <sdma_v4_0_0> (sdma_v4_0)                 │
│ [ 4347.616055] amdgpu 0000:03:00.0: amdgpu: detected ip block number 7 <vcn_v2_6_0> (vcn_v2_6)                   │
│ [ 4347.616070] amdgpu 0000:03:00.0: amdgpu: detected ip block number 8 <jpeg_v2_6_0> (jpeg_v2_6)                 │
│ [ 4347.616074] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: amdgpu_discovery_set_ip_blocks END ===              │
│ [ 4347.616078] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after set_reg_remap ===                             │
│ [ 4347.616081] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after pcie registers ===                            │
│ [ 4347.616084] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after uvd_ctx registers ===                         │
│ [ 4347.616086] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after didt registers ===                            │
│ [ 4347.616089] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after gc_cac registers ===                          │
│ [ 4347.616091] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after se_cac registers ===                          │
│ [ 4347.616094] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: before get_rev_id ===                               │
│ [ 4347.616098] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after ip_version switch ===                         │
│ [ 4347.616101] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: after sriov check ===                               │
│ [ 4347.616103] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: Resizing BAR before VBIOS read ===                  │
│ [ 4347.616106] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: Forcing rebar for Aldebaran ===                     │
│ [ 4347.616114] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: Using max BAR size 16 ===                           │
│ [ 4347.616119] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: Trying ROM BAR for Aldebaran ===                    │
│ [ 4347.616122] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: amdgpu_read_bios START ===                          │
│ [ 4347.616149] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: amdgpu_read_bios after pci_map_rom size=10240 ===   │
│ [ 4347.621223] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: amdgpu_read_bios after memcpy ===                   │
│ [ 4347.621227] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: VBIOS first 64 bytes ===                            │
│ [ 4347.621230] VBIOS: 00000000: 55 aa 14 00 00 00 00 00 00 00 00 00 00 00 00 00                                  │
│ [ 4347.621232] VBIOS: 00000010: 00 00 00 00 00 00 00 00 dc 04 00 00 00 00 49 42                                  │
│ [ 4347.621234] VBIOS: 00000020: 4d d5 00 00 00 00 00 00 00 00 00 00 00 00 00 04                                  │
│ [ 4347.621236] VBIOS: 00000030: 20 37 36 31 32 39 35 35 32 30 00 00 00 00 00 00                                  │
│ [ 4347.621238] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: check_atom_bios START ===                           │
│ [ 4347.621240] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: check_atom_bios after size check ===                │
│ [ 4347.621243] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: check_atom_bios after signature check ===           │
│ [ 4347.621245] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: check_atom_bios after header start ===              │
│ [ 4347.621247] amdgpu 0000:03:00.0: amdgpu: Fetched VBIOS from ROM BAR                                           │
│ [ 4347.621250] amdgpu: ATOM BIOS: 113-D6520100-100                                                               │
│ [ 4347.621260] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: SMUIO_MCM_CONFIG=0x7cf2 TOPOLOGY_ID=31 XGMI=1 ===   │
│ [ 4347.622020] amdgpu 0000:03:00.0: amdgpu: Trusted Memory Zone (TMZ) feature not supported                      │
│ [ 4347.622040] amdgpu 0000:03:00.0: amdgpu: PCIE atomic ops is not supported                                     │
│ [ 4347.622047] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: Initializing SDMA before GPU POST ===               │
│ [ 4347.622314] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: PSP initialized, initializing SDMA ===              │
│ [ 4347.622333] amdgpu 0000:03:00.0: amdgpu: === CRASH PROBE: SDMA initialized, starting GPU POST ===             │
│ [ 4347.622335] amdgpu 0000:03:00.0: amdgpu: GPU posting now...                                                   │
│ [ 4367.623082] [drm:atom_op_jump [amdgpu]] *ERROR* atombios stuck in loop for more than 20secs aborting          │
│ [ 4367.623434] [drm:amdgpu_atom_execute_table_locked [amdgpu]] *ERROR* atombios stuck executing 2488 (len 64, WS │
│ 0, PS …                                                                                                          │
│ [ 4367.623704] amdgpu 0000:03:00.0: amdgpu: gpu post error!                                                      │
│ [ 4367.623707] amdgpu 0000:03:00.0: amdgpu: Fatal error during GPU init                                          │
│ [ 4367.623711] amdgpu 0000:03:00.0: amdgpu: amdgpu: finishing device.                                            │
│ [ 4367.624310] amdgpu 0000:03:00.0: probe with driver amdgpu failed with error -22
1 Like

There might be something wrong with my card, I am on PCIe Gen3 but its making a successful link-up but I never see PCIe IDs :*(

2 Likes

I tried it initially on an older motherboard (Intel 4700k, Z710-A mb) and didn’t see anything when I ran lspci. I would assume this is PCIe 3

This is running now on an AMD 3900X X570, which has PCIe 4.

2 Likes

Have you looked at the voltage rails with an oscilloscope during startup? I’m wondering if the voltage rails are sagging and causing the post error?

1 Like

:person_shrugging: No I don’t have a scope, but the OAM board has the blue status LED lit. That’s the same as the one that lit when I dumped the SPI chip. And the fact that I can read the vbios from the board. I’m mostly proceeding from the software side to poke and prod.

If the power rail voltage droops when the GPU die fires up you can get a brownout. It would probably be less than a millisecond transient and would be imperceptible to the human eye looking at the LED. There’s a Wikipedia article titled ‘Voltage droop’, second image illustrates what I mean.

Below is a AI summary of that I think explains my theory close enough.

Summary

The Anatomy of the Crash

When the Linux kernel loads the amdgpu module, it attempts to perform a GPU POST (Power-On Self-Test), which is essentially booting the GPU. To do this, the driver relies on AtomBIOS—a command-table-driven interpreter embedded in the GPU’s firmware.

  1. amdgpu: GPU posting now...: The driver commands AtomBIOS to start initializing the GPU dies, HBM stacks, and voltage regulator modules (VRMs).

  2. atombios stuck executing 2488...: AtomBIOS executes a specific command table (in this case, table 2488). During POST, these tables are responsible for ramping up voltages, setting base clock frequencies, and training the memory.

  3. atombios stuck in loop for more than 20secs aborting: This is the critical line. When AtomBIOS tells the VRMs to ramp up to a specific power state, it writes to a hardware register and then enters a while loop, waiting for the hardware to flip a specific bit confirming “voltage stable” or “clocks locked.”

Why the Voltage Droop Theory Fits Perfectly

If the custom carrier board experiences a massive voltage sag the millisecond the MI250X tries to wake up its HBM and compute dies, the onboard logic will brown out.

Because the hardware is starved of power, it never successfully reaches the target power state, and it never flips the “ready” bit in the register. AtomBIOS sits blindly in its loop waiting for a hardware response that is never coming. After 20 seconds, the Linux kernel forcefully kills the loop, aborts the POST process, and unloads the driver with error -22 (Device/Argument invalid).

BTW, I think the work you’ve been doing on this is awesome!

3 Likes

I think the issue could be just this simple (from AMD’s documents):

”The two GCDs on an OAM are linked by a high-bandwidth on-package AMD Infinity Fabric™ interconnect, and each OAM exposes additional xGMI ports for direct GPU-to-GPU connectivity across a 4-OAM all-to-all mesh. A typical qualified configuration hosts 4 MI250 OAMs (8 GCDs total) per node.”

The only valid configuration for the MI250 / MI250X is a fully populated node of modules. You can’t initialize just a single module, so the chip is never initialized and is left hanging. You need a full config of 4 modules or you need to do some really major modifications to how the module works (VBIOS modifications).

2 Likes

This was my assumption, but i had hoped we could fiddle the driver past it or trick the OAM into ignoring its defaults. For that I need to build an interposer board. @matthock or @TsuKitsune can you share the pcb layouts?

1 Like

Huh. So Kinda stuck getting 4 cards until someone finds a way around it.

Which, PCIe Variants of the MI2xx cards could show how that’s done. I wonder if anyone here has a PCIe M2xx variant they could pull some dumps from?

@wendell :smiley:

I can extract the MI 210 BIOS from the official AMD update package if you want.

I think there’s a valid configuration of xgmi that can be just a single node.

MODE_X represents that X GPUs will be in the same group, linked together:
MODE_1 (one GPU in a group)
MODE_2 (two GPUs in a group)
MODE_4 (four GPUs in a group)
MODE_8 (eight GPUs in a group)

My thinking is that groups of 1, 2, 4 or 8 are perfectly legal.

I’m headed down another rabbithole though for now, the work on the X570 has come to an end because it’s lacking PCIe atomics support. Dug out a dual EPYC 7K62 and proceeding with that.

If you scroll down it lists “XGMI Support per GPU Model”. Where AMD Instinct MI210X is only listed as: “Between 4/8 GPUs”. So maybe a configuration of 2 modules (since each module is two GPUs) would work.

This might be true actually, because there is no system with more then 8 MI 250x per node, so they must be talking about individual GCDs which the MI 250x has two.

I know the HPE Cray EX235a has 8 GPUs per blade, but there are two nodes per blade, so they are not connected directly onto the same motherboard.

There might be a way to proceed further by skipping the wait (same way the driver already does for SR-IOV VFs on those PSP mailbox regs like C2PMSG_33), which will stop the 20s hang and let the module load instead of dying with -22.

But it won’t actually make the card work. The ATOM post sequence that’s spinning is the part that sets up the real hardware state via PSP. Skip it and you typically end up with locked BARs, no VRAM access, or it just falls over later.

This would require even further driver modifications.

I think the next steps are going to be in the vbios, understanding it better and essentially executing during the asic init stage.

I’m super new to this area so I’m spinning up slowly.

Hello everyone,

I recently brought a supermicro aom-mcm-q baseboard and HPE MI250X GPU. I have a regular server motherboard - ASRock Rome8-2T.

I also bought other parts to power and connect the baseboard to my motherboard.

I use Mean Well RSP 2000W 48V to power the baseboard and GPU. I also use 2x MCIO 174pin to two slimsas x8 and 2x slimsas to PCI-E adapters to connect the baseboard with GPU to my motherboard.

When I first turn on the baseboard using the 48V PSU, it starts with 25W power draw. This is the power for the PSU itself. Then, when I turn on my motherboard, the GPU shows a blue light and the baseboard shows a green light for a few seconds. During this moment, baseboard’s power draw goes up to 30W.

However, when the motherboard reaches the boot screen and starts loading the OS (Ubuntu), the lights will be off on the baseboard and the GPU. Power draw goes back to 25W.

I checked the dmesg and lspci. There were no devices detected on those PCI-E slots. I tried PCIe gen4 and gen3. I also tried recommended BIOS settings from instinct docs amd com/projects/system-acceptance/en/latest/gpus/mi250.html#bios-settings

But still, the device is not powering on and the motherboard cannot detect it. I want to ask if anyone here was able to connect mi250x GPU with supermicro baseboard to their motherboard? If your system is detecting the GPU and showing it in dmesg or lspci, can you please share how you did it?

I am attaching the moment when the baseboard and GPU had lights on.

4 Likes

Quick update.

I made some progress. Using the AMD bios settings webpage instructions and after making a grub change below, I was able to power on the GPU continuously (not just a blink):

GRUB_CMDLINE_LINUX_DEFAULT=“quiet splash amd_iommu=off pci=realloc=off iommu=pt”

I added only pci=realloc=off iommu=pt commands.

After that, I collected dmesg outputs with and without a GPU. I found one single line that was different:

pci 0000:40:01.4: broken device, retraining non-functional downstream link at 2.5GT/s

pci 0000:40:01.4: retraining failed

I will investigate more during this week after work. I think my motherboard is trying to communicate with the GPU but failing. Lspci does not show the device.

Let me know if anyone here was able to connect the supermicro baseboard to a regular server motherboard and lspci detected the GPU.

1 Like

Nice find.

Suggestion: blacklist amdgpu, try to load it at runtime

Watch

dmesg --follow

And then modprobe amdgpu ... with … as your settings. (any kernel options like iommu still need to be on the cmdline). I’d recommend adding loglevel=7 so you get more debug

This will let you see everything logged in an manner more isolated from the startup.

What are you seeing in lspci? Try this (src listing script for transparency)

#!/bin/bash
# mi250-explore.sh - find MI200/MI250X GPUs and show their PCI + sysfs state.
gpu=$(lspci -nn | grep '\[1002:74' | awk '{print $1}')
[ -z "$gpu" ] && { echo "No MI200/MI250X found on this host"; exit 1; }

for dev in $gpu; do
  echo "===== GPU $dev ====="
  lspci -nn -s $dev
  lspci -vvv -s $dev | grep -E 'Control:|LnkSta:|LnkCap:|Region [0-9]:|NUMA node|IRQ|Expansion ROM'
  echo "sysfs:"
  for a in vendor device class irq numa_node enable; do
    echo "  $a = $(cat /sys/bus/pci/devices/0000:$dev/$a 2>/dev/null)"
  done
  echo "  resource:"
  cat /sys/bus/pci/devices/0000:$dev/resource
  echo "  driver: $(basename $(readlink /sys/bus/pci/devices/0000:$dev/driver 2>/dev/null) 2>/dev/null || echo NONE)"
done

Example output for the PCIe OAM card setup I’m currently debugging through


===== GPU 01:00.0 =====
01:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran/MI200 [Instinct MI250X] [1002:7408]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Interrupts: pin B routed to IRQ 11
        NUMA node: 0
        Region 0: Memory at 29000000000 (64-bit, prefetchable) [disabled] [size=64G]
        Region 2: Memory at 2a000000000 (64-bit, prefetchable) [disabled] [size=2M]
        Region 4: I/O ports at 3000 [disabled] [size=256]
        Region 5: Memory at f6300000 (32-bit, non-prefetchable) [disabled] [size=512K]
        Expansion ROM at f6380000 [disabled] [size=128K]
sysfs:
  vendor = 0x1002
  device = 0x7408
  class = 0x038000
  irq = 11
  numa_node = 0
  enable = 0
  resource:
0x0000029000000000 0x0000029fffffffff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x000002a000000000 0x000002a0001fffff 0x000000000014220c
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000003000 0x00000000000030ff 0x0000000000040101
0x00000000f6300000 0x00000000f637ffff 0x0000000000040200
0x00000000f6380000 0x00000000f639ffff 0x0000000000046200
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
0x0000000000000000 0x0000000000000000 0x0000000000000000
  driver: NONE
1 Like

Lspci does not have any entry for this GPU. My server motherboard is not detecting the device at all. I will try to blacklist amdgpu and check the logs of dmesg when I get home.

What adapter are you using to connect the GPU? Is it Nvidia OAM to PCIe board? I think your adapter has the right connection to your motherboard. That is why the GPU is showing up in lspci.

But I will experiment more after work this week. One possible issue is that the cables (mcio to slimsas to PCI-E) I am using are not providing the correct signals to the motherboard. But I will need to check the logs as you suggested.

Thanks!

1 Like