Pushing an agent to modify the amdgpu driver to see how far I can get, and maybe learn a little. Anything with CRASH PROBE: is leftover from a hang/crash I was seeing before.
I managed to coax the driver into reading the vbios from the rom bar. And validated that it matches the vbios I dumped with a clip.
Now I want to caveat that while this looks like progress it sure feels like I’m drunkenly fumbling at a knob and don’t have the key. Looks like a door with a knob alright, I just can’t see the lock.
No I don’t have a scope, but the OAM board has the blue status LED lit. That’s the same as the one that lit when I dumped the SPI chip. And the fact that I can read the vbios from the board. I’m mostly proceeding from the software side to poke and prod.
If the power rail voltage droops when the GPU die fires up you can get a brownout. It would probably be less than a millisecond transient and would be imperceptible to the human eye looking at the LED. There’s a Wikipedia article titled ‘Voltage droop’, second image illustrates what I mean.
Below is a AI summary of that I think explains my theory close enough.
Summary
The Anatomy of the Crash
When the Linux kernel loads the amdgpu module, it attempts to perform a GPU POST (Power-On Self-Test), which is essentially booting the GPU. To do this, the driver relies on AtomBIOS—a command-table-driven interpreter embedded in the GPU’s firmware.
amdgpu: GPU posting now...: The driver commands AtomBIOS to start initializing the GPU dies, HBM stacks, and voltage regulator modules (VRMs).
atombios stuck executing 2488...: AtomBIOS executes a specific command table (in this case, table 2488). During POST, these tables are responsible for ramping up voltages, setting base clock frequencies, and training the memory.
atombios stuck in loop for more than 20secs aborting: This is the critical line. When AtomBIOS tells the VRMs to ramp up to a specific power state, it writes to a hardware register and then enters a while loop, waiting for the hardware to flip a specific bit confirming “voltage stable” or “clocks locked.”
Why the Voltage Droop Theory Fits Perfectly
If the custom carrier board experiences a massive voltage sag the millisecond the MI250X tries to wake up its HBM and compute dies, the onboard logic will brown out.
Because the hardware is starved of power, it never successfully reaches the target power state, and it never flips the “ready” bit in the register. AtomBIOS sits blindly in its loop waiting for a hardware response that is never coming. After 20 seconds, the Linux kernel forcefully kills the loop, aborts the POST process, and unloads the driver with error -22 (Device/Argument invalid).
BTW, I think the work you’ve been doing on this is awesome!
I think the issue could be just this simple (from AMD’s documents):
”The two GCDs on an OAM are linked by a high-bandwidth on-package AMD Infinity Fabric™ interconnect, and each OAM exposes additional xGMI ports for direct GPU-to-GPU connectivity across a 4-OAM all-to-all mesh. A typical qualified configuration hosts 4 MI250 OAMs (8 GCDs total) per node.”
The only valid configuration for the MI250 / MI250X is a fully populated node of modules. You can’t initialize just a single module, so the chip is never initialized and is left hanging. You need a full config of 4 modules or you need to do some really major modifications to how the module works (VBIOS modifications).
This was my assumption, but i had hoped we could fiddle the driver past it or trick the OAM into ignoring its defaults. For that I need to build an interposer board. @matthock or @TsuKitsune can you share the pcb layouts?
I think there’s a valid configuration of xgmi that can be just a single node.
MODE_X represents that X GPUs will be in the same group, linked together:
MODE_1 (one GPU in a group)
MODE_2 (two GPUs in a group)
MODE_4 (four GPUs in a group)
MODE_8 (eight GPUs in a group)
My thinking is that groups of 1, 2, 4 or 8 are perfectly legal.
I’m headed down another rabbithole though for now, the work on the X570 has come to an end because it’s lacking PCIe atomics support. Dug out a dual EPYC 7K62 and proceeding with that.
If you scroll down it lists “XGMI Support per GPU Model”. Where AMD Instinct MI210X is only listed as: “Between 4/8 GPUs”. So maybe a configuration of 2 modules (since each module is two GPUs) would work.
This might be true actually, because there is no system with more then 8 MI 250x per node, so they must be talking about individual GCDs which the MI 250x has two.
I know the HPE Cray EX235a has 8 GPUs per blade, but there are two nodes per blade, so they are not connected directly onto the same motherboard.
There might be a way to proceed further by skipping the wait (same way the driver already does for SR-IOV VFs on those PSP mailbox regs like C2PMSG_33), which will stop the 20s hang and let the module load instead of dying with -22.
But it won’t actually make the card work. The ATOM post sequence that’s spinning is the part that sets up the real hardware state via PSP. Skip it and you typically end up with locked BARs, no VRAM access, or it just falls over later.
This would require even further driver modifications.
I recently brought a supermicro aom-mcm-q baseboard and HPE MI250X GPU. I have a regular server motherboard - ASRock Rome8-2T.
I also bought other parts to power and connect the baseboard to my motherboard.
I use Mean Well RSP 2000W 48V to power the baseboard and GPU. I also use 2x MCIO 174pin to two slimsas x8 and 2x slimsas to PCI-E adapters to connect the baseboard with GPU to my motherboard.
When I first turn on the baseboard using the 48V PSU, it starts with 25W power draw. This is the power for the PSU itself. Then, when I turn on my motherboard, the GPU shows a blue light and the baseboard shows a green light for a few seconds. During this moment, baseboard’s power draw goes up to 30W.
However, when the motherboard reaches the boot screen and starts loading the OS (Ubuntu), the lights will be off on the baseboard and the GPU. Power draw goes back to 25W.
I checked the dmesg and lspci. There were no devices detected on those PCI-E slots. I tried PCIe gen4 and gen3. I also tried recommended BIOS settings from instinct docs amd com/projects/system-acceptance/en/latest/gpus/mi250.html#bios-settings
But still, the device is not powering on and the motherboard cannot detect it. I want to ask if anyone here was able to connect mi250x GPU with supermicro baseboard to their motherboard? If your system is detecting the GPU and showing it in dmesg or lspci, can you please share how you did it?
I am attaching the moment when the baseboard and GPU had lights on.
I made some progress. Using the AMD bios settings webpage instructions and after making a grub change below, I was able to power on the GPU continuously (not just a blink):
After that, I collected dmesg outputs with and without a GPU. I found one single line that was different:
pci 0000:40:01.4: broken device, retraining non-functional downstream link at 2.5GT/s
pci 0000:40:01.4: retraining failed
I will investigate more during this week after work. I think my motherboard is trying to communicate with the GPU but failing. Lspci does not show the device.
Let me know if anyone here was able to connect the supermicro baseboard to a regular server motherboard and lspci detected the GPU.
Suggestion: blacklist amdgpu, try to load it at runtime
Watch
dmesg --follow
And then modprobe amdgpu ... with … as your settings. (any kernel options like iommu still need to be on the cmdline). I’d recommend adding loglevel=7 so you get more debug
This will let you see everything logged in an manner more isolated from the startup.
Lspci does not have any entry for this GPU. My server motherboard is not detecting the device at all. I will try to blacklist amdgpu and check the logs of dmesg when I get home.
What adapter are you using to connect the GPU? Is it Nvidia OAM to PCIe board? I think your adapter has the right connection to your motherboard. That is why the GPU is showing up in lspci.
But I will experiment more after work this week. One possible issue is that the cables (mcio to slimsas to PCI-E) I am using are not providing the correct signals to the motherboard. But I will need to check the logs as you suggested.