[SOLVED] Server Just Died; Need Replacement ASAP

Hardware Hub
, ,
21

I’ll post additional updates just for posterity’s sake again tomorrow/this weekend (USB booting NVMe) and again after I get the new Dell box and RAM, but just want to leave off here thanking @twin_savage, @diizzy, @EniGmA1987, and @skunkfunk for looking out during a stressful time. :heart:

3 Likes
22

Just because it was never detailed…

Old Server: The One That Died

Type Item Price
CPU Intel Core i5-8400 2.8 GHz 6-Core Processor Purchased For $179.00
CPU Cooler Noctua NH-D15S 82.52 CFM CPU Cooler Purchased For $79.90
Motherboard ASRock Z370M Pro4 Micro ATX LGA1151 Motherboard Purchased For $129.99
Memory Corsair Vengeance LPX 16 GB (2 x 8 GB) DDR4-2133 CL13 Memory Purchased For $47.99
Memory Corsair Vengeance LPX 16 GB (2 x 8 GB) DDR4-2133 CL13 Memory Purchased For $123.99
Storage Crucial P3 Plus 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive Purchased For $200.00
Storage Crucial P3 Plus 4 TB M.2-2280 PCIe 4.0 X4 NVME Solid State Drive Purchased For $219.99
Case Fractal Design Define Mini C MicroATX Mid Tower Case Purchased For $79.99
Power Supply SeaSonic FOCUS Plus 550 Gold 550 W 80+ Gold Certified Fully Modular ATX Power Supply Purchased For $79.90
Total $1,140.75

Prices are whatever I paid for the part however many eons ago. The 2xNVMe drives and the cheaper 2x8GB kit were the only things bought this decade (202X), iirc.

After a lot of troubleshooting (not worth detailing here), I can only assume that the board itself is completely toast. C’est la vie. It’s possible it’s the CPU…but I’m willing to gamble on the ASRock board being the failure point than the 8th gen Intel CPU. :man_shrugging:t2:

Lastly: that case has been used for probably almost a dozen builds (depends how you count a Build of Theseus) over the years…it’ll be kind of sad to no longer use it. :crying_cat_face:

What Might’ve Been

This is a WIP look at what I was planning on before the build above shit the bed…

Type Item Price
CPU Intel Core Ultra 9 285K 3.7 GHz 24-Core Processor $573.71 @ Amazon
Case Fractal Design Define 7 Compact ATX Mid Tower Case $135.30 @ B&H
Case Fan Noctua NF-A12x25 G2 PWM 63.15 CFM 120 mm Fan $37.78 @ Amazon
Case Fan Noctua NF-A12x25 G2 PWM 63.15 CFM 120 mm Fan $37.78 @ Amazon
Case Fan Noctua NF-A12x25 G2 PWM 63.15 CFM 120 mm Fan $37.78 @ Amazon
Custom ASUS Pro WS W880-ACE SE (ATX) $449.99
Custom Noctua NM-i17xx-MP83, Mounting Kit for Intel LGA1851 $9.95
Custom 1x48GB 5600MT/s CL46 ECC UDIMM $205.00
Prices include shipping, taxes, rebates, and discounts
Total $1,487.29

When it comes to buying 285 or 285K, the K-version always seems to be cheaper. I had no intention of using its full power.

The Actual New Server

This is what I actually bought: Dell Pro Max Micro Desktop:

  • Configuration Link (don’t know if this link works forever)
  • But basically: did the bare minimum possible build I could to save $ except I upgraded the CPU to an Ultra 9 285
  • 17% discount
  • Price after taxes, shipping, discount, etc.: $1,241.51
  • I wish they’d let me save even more money and order it without the RAM or SSD…but whatever. :stuck_out_tongue:

Then, for RAM: NEMIX RAM 64GB (2X32GB) DDR5 5600MHZ PC5-44800 2Rx8 1.1V CL46 262-PIN ECC Unbuffered SODIMM Memory KIT = $349.99

Brings the total to: $1,591.50.

All-in-all…could’ve been worse, I guess. Certainly cheaper than the Ryzen AI MAX+ stuff I was looking at…and I get my ECC! — Definitely did not have this budgeted for end of year expenses, but I’ll figure it out. Could’ve saved a fortune and done the “buy random LGA1151 boards and see what works and return the rest” but YOLO, I guess. :sunglasses:

Notes on RAM

  1. I know NEMIX is probably sketchy…but this is what memtest is for. :man_shrugging:t2: — I couldn’t find anything else less sketch without nearly (or more than!) doubling my spend…
  2. I could’ve spent a little more to see if the Dell box could do 2x48GB for 96GB…but I just wanted to go for what was in spec for now, plus…this’ll already be double the 32GB I had before.

What about bulk storage‽

I touched on it earlier, but I have a 10-bay Sabrent thing I got on sale for a bangin’ deal fairly recently.

  • In there I have 2xSeagate Exos ST24000NM000C 24TB (manufacturer recertified) I picked up for $279.99/each recently.
  • I also have 8xSeagate EXOS ST16000NM001G 16TB purchased all at once brand new well over 5 years ago for $384/each.
    • These drives were in SHR2 in a Synology box for years before I finally moved my storage and compute onto the same machine (but still technically 2 boxes with a type-c cable :stuck_out_tongue:).

It’s all thrown together in a ZFS array of mirrors. Sure, I lose half my capacity, but:

  • Expanding my storage also takes only 2 drives and it’s stupid fast to do so.
  • Scrubs are stupid fast.
  • The array itself is stupid fast, in general.

Sensing a theme? :stuck_out_tongue: — It’s also, in theory, possible for me to lose up to 5 drives without having to restore from a backup, with the caveat being I can’t lose 2 in the same mirror. I weighed my options and went with this. :man_shrugging:t2:

Anyway, I think I’ve spent enough time rambling about something that’ll probably go unread. :joy_cat: — No server to mess with, so lots of free time. :crazy_face:

1 Like
23

Update: limping along on that i5-4670K and one of my NVMe boot drives connected via USB. ZFS array and everything else seems to basically be working!

Obviously my md arrays are in degraded states with 1 drive missing, but whatever.

There’s also 1 random docker container I have that isn’t working, but it’s unimportant. I’ll take the win goddamnit. :smile:

1 Like
24

I solved the one container issue, so I guess we’re “golden.”

The i5-4670K is about 40% slower than the i5-8400[1]…and boy can I feel it lol

  1. https://www.cpubenchmark.net/compare/3097vs1921/Intel-i5-8400-vs-Intel-i5-4670K ↩︎

25

You could upgrade the cpu to a 4790k then it would be around 8% slower than the 8400. But use twice as much power probably… :smiley:

2 Likes
26

Not worth the $73 when the new machine is already on the way, but I chuckled. :smile:

Edit: the step back in hardware that hurts the most is losing out on some hardware acceleration for Jellyfin… :grimacing:

27

In a strange turn of events…my Dell box is now supposed to arrive tomorrow (original estimate was 6th-11th), but my RAM still won’t be here until the 6th. :sweat_smile:

28

Now the RAM is on the delivery truck to arrive today and my desktop still shows as in El Paso…we’ll get it all someday! :stuck_out_tongue:

29

Got the server today. Threw in the ECC RAM a short bit ago and loaded up PassMark MemTest86 Pro…just to discover Dell doesn’t support ECC error injection. Oh well. Still worth doing a full test on all the RAM to make sure.

Speaking of ECC…what on Earth is the standard thing to install in Linux now? edac-utils? (Debian)

30

Ah, looks like rasdaemon is the “new” hotness.

31

So does this mean the ECC memory doesnt do any of the ECC things it is supposed to? Or just that you cannot test it properly beforehand on that Dell?

32

I think my kernel is currently hampering me. Trying to figure out what edac module I’m needing and not loading.

Edit:

$ find /lib/modules/6.12.48+deb13-amd64/ | grep -i -E 'edac'
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i3200_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i5100_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/skx_edac_common.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/x38_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/e752x_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/pnd2_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i3000_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/sb_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/igen6_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i7core_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i10nm_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/edac_mce_amd.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/skx_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/amd64_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i82975x_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i5400_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/ie31200_edac.ko.xz
/lib/modules/6.12.48+deb13-amd64/kernel/drivers/edac/i7300_edac.ko.xz

$ sudo lsmod | grep edac
$

Edit 2:

This tells me that EDAC support should be built-in…

$ grep CONFIG_EDAC /boot/config-6.12.48+deb13-amd64
CONFIG_EDAC_ATOMIC_SCRUB=y
CONFIG_EDAC_SUPPORT=y
CONFIG_EDAC=y
CONFIG_EDAC_LEGACY_SYSFS=y
# CONFIG_EDAC_DEBUG is not set
CONFIG_EDAC_DECODE_MCE=m
# CONFIG_EDAC_GHES is not set
CONFIG_EDAC_AMD64=m
CONFIG_EDAC_E752X=m
CONFIG_EDAC_I82975X=m
CONFIG_EDAC_I3000=m
CONFIG_EDAC_I3200=m
CONFIG_EDAC_IE31200=m
CONFIG_EDAC_X38=m
CONFIG_EDAC_I5400=m
CONFIG_EDAC_I7CORE=m
CONFIG_EDAC_I5100=m
CONFIG_EDAC_I7300=m
CONFIG_EDAC_SBRIDGE=m
CONFIG_EDAC_SKX=m
CONFIG_EDAC_I10NM=m
CONFIG_EDAC_PND2=m
CONFIG_EDAC_IGEN6=m

…but:

$ sudo ras-mc-ctl --status
ras-mc-ctl: drivers not loaded.
33

Yeah…I think Arrow Lake simply doesn’t have EDAC support in the Linux kernel yet, not even in 6.18. Source: History for drivers/edac/ie31200_edac.c - torvalds/linux · GitHub

Looks like the newest supported stuff is some 14th gen chips. :grimacing: So…maybe I happen to wake up to EDAC support for my CPU someday lol

34

Recovering this RAID1 setup I had was a little weird…but got there in the end. Just needed to:

  1. Wipe the drive I didn’t boot from and partition it accordingly
  2. Add the new partitions to my existing, degraded arrays
  3. Grow the arrays to 3 mirrors instead 2
  4. Wait for resyncs to finish
  5. Shrink back to 2 from 3 mirrors
  6. sudo update-initramfs -u

Edit: grow to add drive as spare and shrink to promote it to active after resync — some weird idiosyncrasy with mdadm getting upset when a drive is missing too long, I guess

35

In case anyone on the Internet is searching for info on cooling performance for the FCM2250 (Dell Pro Max Micro, 2025) with a Core Ultra 9 285: if you boost the fan by 10% in the BIOS, you can comfortably run 5 of 8 the p-cores at 100% at 5.2GHz 24/7 and stay at 80C or less on the chip. 10% bump (you can go up to 50%) on the fan curve is definitely not quiet, wouldn’t recommend if you’re going to have the computer next to you. Mine’s locked away in an office, so it’s fine.