Can't install or run any OS on RomeD8-2T, EPYC 7343

My dear angry sysadmins,

I have an issue that I cannot install PVE 9.1 or 9.2 or debian 13.5.0 on my new Romed8-2T from asrock rack. I can get it to the installer selection 100% of the time, but somewhere between my selection of the install method and finishing the install, it always crashes and reboots. I have gotten to the end of PVE 9.2 with all settings and it went through the entire installer just fine, except it didn’t: it was frozen on 100% and after 30 mins, I restarted the server. Of course there was nothing to boot, and it restarted back into the installer…


Hardware:

  • EPYC 7343 (previously a 7282 that did not boot with code d0)
  • Micron 2x 32GB DDR4 ECC RDIMM 3200 (MTA18ASF4G72PDZ-3G2E1TI)
    • not on the QVL, but off by ending in TI, instead of UI. and boots, unlike the SKhynix that i had before
  • SeaSonic Prime TX 1300W
  • WD SN850X 2TB [2x] (for VMs)
  • LSI 9400-16i IT-Mode
  • Intenso High (120GB) for boot
  • Silverstone RM-61-312 (if that matters at all)
  • 1x10G LAN and IPMI LAN are connected to the router

I have tried pretty much everything I can think of:

  • IOMMU, above 4G decoding, SRIOV, rebar: off and on
  • CMOS reset multiple times
  • Reseated cpu
  • reseated RAM
  • memtest (from the PVE iso): 100% pass, 0 errors
  • several install methods:
    • USB via fedora mediawriter on 3.0 and 2.0 stick
    • Motherboard IPMI
    • ventoy
  • checked checksum for both PVE 9.1 and 9.2, both pass
  • secure boot is off
  • install with nomodeset and normal setting
  • graphical install and terminal install
  • updated Bios to newest, 4.10
  • everything is unplugged, save for one stick of ram and one sata ssd
  • logs don’t tell me anything but I am also stupid noob

I have been beating my head against this rock wall for the past 3 days. No matter what I do I get the same result: a crash somewhere between selecting the installer and finishing it. usually it happens before the proper installer UI even opens.

Here are some examples of the installer crashing/failing/whatever you want to call it. These are “more successful”, as in: I got marginally further in the installer.

Debian video on YT
PVE video on YT

Please don’t judge my hardware choices, a lot of it are things I already had or had access to. Berate me on the dual 120gb boot ssds if you wish.

PS:

I purchased most of this hardware 6 weeks ago and had issues with the RAM and CPU already. I thought I could finally catch a break but this project seems to just be cursed.

Also, I am not a complete noob, as I do have a TrueNAS core server and daily drive Linux since over a year (CachyOS & distrohopping on my FW13 Laptop, currently fedora KDE). I know how to install a damn OS, unless it does not let me of course.

I have tried installing PVE on another machine and moving the ssd to my romed8. That also crashes once it start to boot.

I have disabled low power c states, that i could find in the bios

1 Like

Try changing the NUMA setting. It’s hard required with very recent kernels as far as I can tell. But it does not come up often.

1 Like

I will try that but freeBSD and Windows won’t install either, so the linux kernel is irrelevant in those cases.

Yeah, so NPS1 (default) fails obviously. Auto, NPS2 and NPS4 fail too. Anything else i can set in the memory settings (see image)?
keep in mind that I only have one stick in there right now

Asrock Rack is quite unhelpful right now, I was hoping they could tell me something.

I have also updated to BIOS 4.30 and BMC 3.08, which is the most current. Made no difference.

With the auto NUMA setting I can get into the installer, but I was able to complete the installer previously too, so that is no progress in my eyes. (it went to 100% but froze and did not install anything afaict)

here is another example but with PVE preinstalled on the btrfs mirrored ssds (which works perfectly fine on my desktop)

Maybe someone knows more about those errors at the start

If it does this for any OS, even one already installed, I would then next suspect:

  1. The disk interface/cable/controller. A bad cable, bad/overheating SATA/SAS controller or faulty backplane can cause no end of trouble. To test this you’d need an nvme drive or try different slots/cables, or even a usb adapter (which is it’s own can of worms they can be finicky, and often need a heatsink added to the adapter controller)
  2. The PSU. If the power supply is bad, this can also manifest as weird problems. Only way to diagnose that those is to have a known good one handy and be able to hook it up to the system.
1 Like

Also, try dropping the memory speed a step down to 2933 or the next lower.

Thanks, Log.
I am at work now so I can’t test it yet but about the drives:
1a) This crash also happens with no drives attached, I can try unplugging the cable MoBo side as well.
1b) It is not the backplane since the drives are direct sff8643 to Sata, with the Asrock provided cable. I can test with the second cable that came with it, but I do not see it making a difference, because it crashes with no drives as well.

  1. the PSU could be the issue of course but those glitches seem to be more inconsistent from what I know. Worth a try nonetheless (it will be a pinta to get the other one out of my desktop tho)

  2. if memtest passes, dies lowering the speed make much difference? I understand that memtest is not perfect and that we don’t trust any part of the equation, but I do not know the confidence delta between memtest passing and memory/mem controller working.

Honestly I’m grasping at straws myself, basically adjusting the memory speed down is just a quick and easy thing to eliminate.

I typically use Passmark’s Memtest86 (which has ECC reporting) to test my ECC memory, and in my experience it does a fantastic of reasonably quickly revealing the sort of instability which can’t be ignored. I would normally expect the memory operations that come from running/installing a linux OS to be close enough to the memtest tests, that a problem would be revealed, but error generation from workloads can be very weird sometimes.

What’s strange is you can run proxmox’s Memtest86+, but no other OS. At this point it sounds like an issue with some setting in the bios, but I’ve no idea what would cause this, and updating your bios should have reset things into a reasonable default. It’s possible that proxmox’s Memtest86+ cannot see ECC errors being reported on your system, so you could be getting a bunch silently when doing the test, and then when running a real OS the errors escalate further into multi-bit errors.

Other things to try:

  1. Checking the voltage of the CMOS coin cell.
  2. Checking that the misc jumper headers are in a reasonable state. For example the multiple NVDIMM Power Support Jumpers probably shouldn’t be enabled.
  3. Check if the watch dog timer is enabled in bios. I believe this can attempt to reboot the system if it somehow detects that a chosen kernel hasn’t booted correctly. Not sure how this works.
1 Like

even checking your 12/5/3.3 PSU rails(if they are all present) could show an issue. Seasonic makes a good PSU, but you never know

@Log nailed what I would cover, a random jumper or dead CMOS battery. Even testing with assorted sticks? I don’t know your availability of spares

1 Like

Thanks for both your input.
Once I am back from work, I will try those options. The IPMI did not complain about cmos battery being bad, it did notice when it was missing, so the sensor works at least kek.

What is the voltage that a charged cmos cell should have?
I will try swapping with my desktop, if i need to try a different PSU, I can take the cmos out as well.

I bought other ECC RDIMMs first. From skhynix, they did not work. Code 16 if i recall correctly, not documented in the manual…
Also tried known working standard non ecc DIMMs (UDIMM, right?), those gave me code 16 as well. It seems the board (or cpu) is quite picky with memory. My desktop’s corsair vengeance RAM kit is the last one I can try, but I am not expecting it to work.

About checking the rail: should I unplug the psu and jump the pwr btn & check with multimeter or are the motherboard’s IPMI report good enough?

People usually say that the onboard sensors are not to be trusted but that may be more from consumer level stuff, not server mobos. Also that may have gotten better in more recent years

1 Like

CMOS battery voltage is about 3.066V just after pulling it out.

Ok, so while gathering my thoughts in this draft, i tried the 7402P that I just posted about again, and… the system boots with no issues. straight into the already installed PVE on mirrored btrfs ssds. I still need to try it with all 4 sticks but damn

2 Likes