Low Power 3 x 2U kubernetes cluster build (replacing NUCs)

Hardware Hub Build a PC
,
1

Hi folks!

I currently own 3 x NUC13s that I run Talos Linux on. I also have a set of Sonnet Solo 10G thunderbolt NICs attached to them (because I run ceph via rook-ceph).

These all sit in a 25U rack with a UPS, a 36 bay Supermicro DAS + NAS, UPSes, the whole 9 yards.

The cluster of i5 NUCs is my main compute and I’m not happy with temps and IO; basically I have the space in the rack and I think I can do better heat dissipation and get more IO for not that many extra watts.

My plan is to get something like the ASRock Rack D1749NTD4U-4T4O or similar, stick it in a 2U case with an appropriate PSU, fans, and an Intel Arc 310 for transcoding. Obviously I’d get three of these.

My requirements are:

  • low power cpu (transcoding optional since I can get an Arc)
  • 128GB RAM support
  • dual 10GBE NIC prefered
  • 2 x M.2 slots
  • at least 3 x PCIE ports, the higher the speed the better
  • enterprise gear (Supermicro, Asus Rack, Gigabyte Rack, Asrock Rack) prefered
  • BMC

I still need to pick a 2U rack mountable chassis. I only have 8U available in the rack so best I can do is 3x2U or 2x2U + 1x4U (don’t mind going heterogenous).

I’m just looking for someone to bounce ideas off of, a stochastic parrot can only get me so far (never tells me I’m dumb).

Would this be a good idea and accomplish what I’m trying to achieve? Will the better airflow compensate for the slightly higher power?
Which case(s) should I pick? Any other details that I’m missing?

Thanks.

2

I’m building a similar cluster. 8U (2+2+4), High performance Ceph with U.2 and 25Gbit networking, low power (and thus mostly quiet).
Check out my build log, ongoing project with 2 nodes ready now (virtualized Ceph so far)

I mainly focused on storage and networking and I’m using AsRock EPYC4000D4U board with a Ryzen 7900. Allows for 4x U.2 and 25Gbit NIC, but no room for dGPU for that nice Intel GPU plugin.

It’s really hard to find a good board without going full EPYC 9000, but I found my niche. You really have to compromise on storage hardware to get PCIe lanes to fit in an A310. In my case, that wasn’t that important, so I got away with less.

NUCs don’t have space for U.2 drives. Which is why I didn’t consider Minisforum MS-01 or other similar mini/micro form factors. No space, can’t dissipate the heat, fans (if present) going nuts.

And Ceph really wants/needs Enterprise drives with PLP to get IOPS going. This is really the only de-facto hard requirement for proper IO. And threads, because Ceph is basically CPU bound once you deal with NVMe speeds.

2U has plenty of space for NVMe (I got a Silverstone 2U and used the 5.25" external for IcyDock backplane).

3

Post some data on your current setup, what kind of power consumption you have on idle, low activity and normal load? Eg. what do you consider low power and low enough?

Consumer gear like NUC13s will have much better power draw under idle and low loads than pretty much any enterprise gear, even specialty embedded models.

Unless you plan to consolidate 3 → 1 node , I doubt you will gain any power or barely marginal savings this way.

I learned the painful lesson trying the same thing with epyc platform, still running hp elitedesk as my proxmox host just due to baseline power draw.

1 Like
4

EPYC isn’t EPYC. Sure, if you get 9004/9005, maybe even F-SKU, that’s a problem for the Wattmeter. But EPYC 8000 Siena has some really impressive idle and perf/watt figures (which translates well into noise levels, important for a lot of us). And I’ve seen a bunch of guys here at L1T in the past using EPYC Rome both because cheap and because fairly low power by server standards.
The clock speeds are really the thing that ramps up the power bill. And why I like Siena with Zen4c and below average clocks. Sometimes you just need cores, not frequency, while still having memory channels and PCIe lanes.
And Kubernetes loves cores and memory, clocks are always nice but optional.

Can’t be consolidation into a single machine. All those redundant parts eating power (PSU, memory, CPU, BMC, NIC) is just way more efficient bunched up in a single server. But cluster, Kubernetes and a lot of modern enterprise stuff ask for multiple nodes, HA and that stuff…“Cloud Computing” and “Hyper-convergence” are the buzzwords.

I do understand why there is a big appeal for NUC, MS-01, your HP elitedesk or even SBCs. There is nothing outside of laptops with better perf/W and low idle.

But I also decided against it, because Enterprise NVMe don’t go well with mini/micro chassis and connectivity. Not enough lanes, not enough air to breathe. But they’re just great if the stuff you need fits into that package. And K8s+Rook, if you want performance, will hit the wall. Ceph is great, but Ceph is also a very demanding wife.

5

Sadly most Epyc CPUs are a fair bit too power hungry (quick search says they idle at over 30W easily).
The rack is under the stairs close to my living room and main hangout area so I can’t go too crazy; I’ve got not only the wife approval factor to keep in mind but my own sanity as well haha.

NUCs don’t have space for U.2 drives. Which is why I didn’t consider Minisforum MS-01 or other similar mini/micro form factors. No space, can’t dissipate the heat, fans (if present) going nuts.

And Ceph really wants/needs Enterprise drives with PLP to get IOPS going. This is really the only de-facto hard requirement for proper IO. And threads, because Ceph is basically CPU bound once you deal with NVMe speeds.

I’ve been using Ceph via Rook Ceph in kubernetes for over 5 years and it performs really really well, it’s only lately temps have gone crazy because I kept adding workloads (I have around 200 workloads, I experiment with a lot of stuff).
I wholeheartedly agree you’d ideally want enterprise drives but I’ve been compromising with consumer M.2 for years and it’s worked out fine :tm: . I’m open to suggestions in terms of bang for buck though, I’d obviously love to go PLP.

@greatnull

Post some data on your current setup, what kind of power consumption you have on idle, low activity and normal load? Eg. what do you consider low power and low enough?

Sadly I don’t have exact data for compute alone, but I idle around 400W total (with the JBOD and Supermicro NAS). I would like power draw to not go up more than 50W, mostly because of the heat/noise it brings.

I could go single node but at this point my home lab is production and I need a cluster.

As for MS-01 I’ve thought about it but decided against it after reading so many horror stories with reliability on L1T and basically everywhere else.

I realized the only compute board I’m really happy with is my Xeon D SM X11, the only downside is a single PCIE slot. So I looked around and found the Asrock Rack D1541D4U-2T8R Xeon D or derivations of it, I’m leaning more towards three of those at the moment.

The Asrock Rack D1541D4U-2T8R has 2x M.2, dual 10GBe, 2x PCIE slots, a BMC (if I’m not mistaken). I figured I’d grab a 2U chassis and see if that works and if I’m happy just get two more.

Any issues with the board/setup that I might’ve missed? Also what enterprise drives should I consider to replace my 3x2TB 970 Evos?

6

( have around 200 workloads

my home lab is production and I need a cluster.

and add rook managed ceph into mix, it now starts making sense.

Medium to high usage kubernetes cluster with emphasis on cluster. Rook implies necessity for converging storage and processing nodes.

Well optimizing for idle/low activity power use typical for many homemakers is pointless. You hardware is unlikely to hit low power states due to constant high activity + general cluster maintenance workload.

What is possible is optimize for power under load, which really sucks comparatively for many higher end consumer cpus due to heavy freq boosting.

I.e rightsize using lower end enterprise grade hardware (xeon d and like, epyc embedded, xeon w maybe?).

But you really should measure each of you devices first using kill-a-watt type of device. Or cheap smart plug with energy monitoring, also useful in future.

Like this or this:

7

Indeed! As it happens I have some shelly plugs on the way, I should get a clearer picture in under a week hopefully.

Xeon D is where I’m at for the moment, I just hope I can get an appropriate case so fans don’t go crazy.