EPYC 9575F + ASRock Rack BERGAMOD8-2L2T — instant hard power-off under memory-bound load, no MCE/ECC/SEL/BERT record anywhere

Two weeks into this and I’m out of ideas. Posting in case anyone has seen this failure mode on Turin, or on this board specifically.

Symptom

Under memory-latency-bound load at high thread counts, the machine dies instantly. Not a panic, not a reboot — it goes completely dark and will not POST again until AC is physically removed (PSU switch off or unplug). The power button does nothing until then.

Originally found by running Agisoft Metashape, which killed it unattended. Minimal reproducer:


stress-ng --stream 128 --timeout 300s

Depending on configuration this kills it anywhere from a few seconds to a few minutes. It is probabilistic — some configurations survive for hours and then die, which made this miserable to bisect.

The part I can’t get past

There is no error record. Anywhere. Every reporting path this platform has comes back empty:

  • No kernel panic, no oops, no MCE

  • rasdaemon / EDAC: zero CE, zero UE on every channel, with ECC confirmed active

  • I additionally enabled DRAM Write CRC, Read CRC and RCD Parity in BIOS — still zero

  • BMC SEL: nothing at the time of the crash (only “Power off/down — Deasserted” when power returns)

  • No BERT/ERST record on the following boot

  • netconsole to another box on the LAN captures nothing — verified working with a test message first, so the pipe is good. The kernel is executing normally and then the lights go out.

That reads like a hardware protection latch or fabric halt below the OS, but nothing will tell me what tripped.

Hardware

CPU AMD EPYC 9575F (64C/128T, 400W default TDP)
Board ASRock Rack BERGAMOD8-2L2T, BIOS 10.09, BMC 10.03.00
RAM 8× G.Skill F5-6000R3036G48GE8-G5N (48GB RDIMM, 384GB total)
At SPD default 4800 MT/s, 1.10V — not EXPO, not overclocked
PSU Seasonic PRIME PX-2200
Cooling 360mm AIO
Storage Samsung 990 Pro 2TB NVMe
GPU RTX 5090 (headless box, llvmpipe desktop)
OS Fedora 44 (7.1.7 + 7.1.5), Ubuntu 24.04.4 (6.8), Ubuntu 26.04 (7.0.0-14)

Board has 8 DIMM slots, so 8 of the CPU’s 12 channels are populated. NPS1.

Ruled out

Thermal — 57–60°C under the failing load. (There are CPU_THERMTRIP events in the SEL from the very first boot — that was me forgetting to plug in the AIO fans. Nothing since.)

PSU — 2200W unit, package draws 320–400W. Not close.

GPU — removed entirely, still crashes. There were PCI PERR bursts in the SEL early on; those stopped after a BIOS update plus moving the card to a different slot, and the crashes continued unchanged. Red herring.

Storage — NVMe SMART clean, 0 media errors, btrfs scrub clean.

Memory — this is the one I was most confident about and it’s dead. It crashes with a single DIMM at 3600 MT/s, and passes 900s with all 8 at 4800. Tested 8/4/2/1 modules, multiple different modules, multiple slots. Module count, speed and channel population are all irrelevant. Combined with the zero ECC/CRC counters, the DIMMs are as exonerated as I can make them.

Distro / kernel — fails on Fedora 7.1.7, Ubuntu 6.8 and Ubuntu 7.0.0. The rate varies a lot (6.8 is much more tolerant, 7.1.7 dies in seconds) but nothing is immune.

Power management — pcie_aspm=off, amd_pstate=disable, processor.max_cstate=1, idle=poll, nosmt, and a 320W package cap via HSMP each reduced the failure rate and none of them fixed it. Every one of them looked like the answer for a while, then failed on a longer run.

The discriminator: load type, not power

stress-ng --cpu 128 runs at a flat 400W indefinitely with no problem.

stress-ng --stream 128 kills it at ~320W.

So it isn’t total power. It’s something about saturating the memory path. DDR bandwidth sits at ~255 GB/s (83% of the 307 GB/s the SMU reports as max) during the failing load.

Thread count is the strongest single correlate. Anything running ≤64 active threads has survived 900s runs repeatedly, including all 8 CCDs at one thread per core. 96–128 threads fails.

A secondary observation I can’t fully explain

Using taskset to pin work to specific CCDs (8 cores + SMT siblings each, per shared_cpu_list), at constant 112 threads and constant package power, the failure rate depends on which CCDs are loaded:

  • All CCDs except CCD4 → instant fail, 3/3 attempts

  • All CCDs except CCD3 → pass

  • All CCDs except CCD7 → passed many times; eventually failed once at 300s

I built a whole model on this and then it broke, so I’m presenting it as a rate difference across a limited number of trials, not a proven mechanism. But it’s suggestive: the motherboard has no knowledge of which CCDs are active and supplies identical rails either way, so a VRM/PSU/memory explanation struggles with two runs at identical load and opposite outcomes.

Running with CCD7 isolated (isolcpus=56-63,120-127, 112 threads) has given me 6 days uptime and several completed Metashape jobs, which is by far the best I’ve had. Not a fix, but livable.

Two firmware bugs found along the way

Independent of the crash, flagging for anyone else on this board:

  1. BIOS TDP/PPT does not enforce. Set TDP Control and PPT Control to Manual, value 320 — the menu displays 320, turbostat measures 379W under load. Setting the identical limit via HSMP (/sys/devices/platform/AMDI0097:00/hwmon/hwmon*/power1_cap) works correctly: 319.7W measured, and pwr_current_active_freq_limit_source reports PPT Limit. So the SMU is fine and the BIOS path is broken. (It also only took effect at all after a full AC power cycle — a warm reboot silently discarded it.)

  2. Platform First Error Handling = Disabled has no effect. GHES: APEI firmware first mode is enabled by APEI bit still appears at boot and _OSC reports platform retains control of PCIe features (AE_SUPPORT). Presumably a large part of why nothing ever gets logged.

The thing that’s nagging me

ASRock’s release notes for BIOS 10.02 and 10.09 both state: “Only supports Turin CPU below 400W” and “Only supports DRAM frequency max. up to 4800MHz”. The 9575F is a 400W default TDP part. Their CPU support list shows the 9575F as supported from 10.02 onward.

Also — the BERGAMOD8-2L2T manual (v1.0, July 2023) lists support for EPYC 9004 and 97x4, i.e. Genoa and Bergamo. Turin support came later via BIOS. ASRock has since released the TURIND8-2L2T, which STH commenters note appears externally identical to this board, with speculation that the changes were for memory signal integrity.

Questions

  1. Is anyone running a 400W Turin SKU on a BERGAMOD8-2L2T successfully under sustained memory-bound load?

  2. Does anyone know what actually changed between the BERGAMOD8-2L2T and the TURIND8-2L2T?

  3. Given PFEH won’t disable and firmware-first won’t relinquish, is there any way to make this platform say what tripped? I’ve tried pcie_ports=native and hest_disable — the kernel takes over MCA decoding and still sees nothing.

  4. Does “fails at 96+ threads, stable at ≤64, no error records at any layer, memory eliminated” read as CPU or board to you? I can’t swap either part to find out.

Happy to run anything useful. Thanks for reading.

1 Like

Currently my PC w/ AMD zen 3 (tried w/ 5900xt & 5500 cpus) will appear to hang. What actually happens is the GPU requires a power cycle before display will work again. The machine actually reboots, but the GPU has no display!

Have you tried to SSH into it after the failure but before power cycling?

My machine is resetting due to infinity fabric flood. I only figured this out after I realized I was looking at the wrong boot session in journalctl.

Good luck.

I only have older Epyc Milan systems and looks like you’ve been very thorough already but a couple of ideas that I didn’t see mentioned:

  1. Did you try fans over the DIMMs? DDR5 RDIMMs can chew quite a bit of power on their own and get hot. I don’t know if there would be thermal monitoring that would trip but I’ve seen RAM temperature issues before.

  2. Did you try a different known-good PSU? Even big PSUs go bad and can introduce strange effects. Could try monitoring the various voltages from the BMC web UI to see if anything fluctuates between idle and memory-loaded tests. Won’t help if it’s a brief transient but might show something.

  3. Did you try without a GPU? You should have VGA from the ASPEED BMC, or use the remote control from the BMC interface from another machine.

I have had an Epyc do this before. Instant power off. No warnings in BMC, no log messages, and requires a full power off/on to get it back up.
It was a bad chip. I replaced it and its been stable for months. Thats not the only reason this kind of thing could happen, though. There could be a weird power delivery issue on the motherboard. Or some other thing triggering the psu to insta-chop current.

1 Like

^ this. I’ve also seen this when VRM was overheating but there were clues in the IPMI. If you have a thermal camera point it at the motherboard. Dump a bunch of air over the motherboard? In our case we think it was one singular a marginal VRM that was much hotter than the others as doing a board swap fixed it.

Have seen this from chips too, though. You can cTDP on the fly and drop the TDP by 50w and see if that fixes it. If it does, and its not thermal, there’s your issue – the chip.

Thanks all, some very useful pointers.

@thr3e that’s the closest match I’ve seen to my symptom. Encouraging
(in a grim way) that it turned out to be the chip.

@wendell I think I’ve effectively already run that test, with a partial
result:

I capped package power to 320W (80W below stock) via HSMP, verified
enforced, turbostat reads 319.7W and pwr_current_active_freq_limit_source
reports “PPT Limit”. It reduced the failure rate but did not
eliminate it; runs still eventually died. (Note the BIOS TDP/PPT
setting on this board is broken, it displays 320 but measures 379W under
load, so the cap has to go through HSMP to actually apply.)

Three things that I think argue against VRM/thermal here:

  1. stress-ng --cpu 128 pulls a flat 400W indefinitely with zero problems,
    more power, more current and more VRM heat than the load that kills it.
    stress-ng --stream 128 kills it at ~320W. The failure happens at LOWER
    power than a load that runs forever.

  2. It frequently dies within a second or two of starting the load, and it
    does so identically from a cold boot or after hours of uptime. Nothing
    is heating up in that time.

  3. CPU sits at 57-60°C under the failing load. I did have a thermal camera
    on the board during testing (I was looking at the DIMMs at the time, so
    no saved image of the VRM area) and don’t recall anything around the
    socket standing out, the only notably warm thing was the 10G NIC.

@xzpfzxds

  1. Haven’t tried forced DIMM airflow, but it also crashes with a single
    DIMM installed at 3600 MT/s, which makes DIMM thermals hard to blame.
  2. Can’t swap the PSU, no spare, and it’s a 2200W unit running at
    320-400W. I’ve polled VOLT_P0_VSOC / VOLT_VDD_11 / VOLT_VDDIO from the
    BMC at 1Hz during runs with nothing moving before the crash, though as
    you say that won’t catch a fast transient.
  3. GPU is out, removed entirely, fault persists. Running off the ASPEED.

One more data point that I think argues against board-side power delivery:
using taskset at a constant 112 threads and constant package power, whether
it crashes depends on WHICH CCDs are loaded. “All CCDs except CCD4” fails
instantly, 3/3 attempts; “all except CCD3” passes. Same thread count, same
memory pressure, same current draw, the motherboard has no knowledge of
which CCDs are active and supplies identical rails either way.

RMA request with the CPU vendor is already submitted.

1 Like