Struggling over homelab rebuild roadmap

Hardware Hub Networking
,
1

At a bit of a crossroads in how to approach a homelab rebuild/upgrade.

For context, I downsized my homelab to an AIO solution several years ago, picking up a 5-bay NAS from a popular brand. My approach was to simplify as much as possible, and I have no room for a full-sized rack any longer. 10" racks don’t interest me. This is more functional than it is hobby, so I want to remain as lean as possible, but with a good amount of power, reliability, etc.

This aforementioned NAS manufacturer is slow to release new products, leaving me hitting the ceiling on that device so I’m keen to move to a different solution. Unfortunately, it’s been several years and I’m terribly out of the loop.

My goal is to move as many of the services I’ve been forced to seek elsewhere back to self-hosted solutions, serve media, and have an available, performant, and easy to access development environment that I can access both locally, and whilst traveling.

Brief summary of requirements:

  • Self-hosting (cloud-syncing, note-taking, documentation, automation, hosting development builds, DNS, VPN, etc.)
  • Media server with automation (little to no external access needed)
  • Development environment (Python, React, PHP, databases, version control)

Current setup:
5-bay NAS
Nighthawk AXE11000
16-port Gigabit (unmanaged) Switch
ARRIS SURFboard SB8200 Modem

I’m considering one of two paths:

AIO route:
45HomeLab HL8 w/ upgraded CPU, ram, SSDs/HDDs (click all the buttons)
Running Unraid or TrueNas

Mini PC route
Keep the current NAS as big, dumb storage
3x Minisforum MS-01s w/ HA (96gb ram, 1-2tb SSDs, no dGPU; haven’t decided between 12th or 13th gen just yet – I think the added performance of 13900H are marginal)

For the latter, I’d strongly considered starting with a single MS-01; while I have some Linux/UNIX competency and I’m not afraid to dig in and keep the setup maintained, I’d like to reduce complexity as much as possible. HA, however, is extremely desirable considering my above requirements.

With the HL8, I just haven’t seen enough from the community and not sure of raw performance; again, being out of the loop I’m nervous I could potentially hit a ceiling and will be stuck with an overpowered NAS and end up going with a mini-PC if/when I learn that hard lesson.

Any insights, opinions, or admonitions are appreciated.

1 Like
2

I would stay far away from the cheap chinese “mini pcs” given that reliability and aftermarket support (bios etc) is poor at best and “big.little” CPUs seems to perform poorly in virt envs. There are also already a bunch of recent threads that pretty much answers all these questions.

3

You asked? :nerd_face:

How about building your own?

  • 45Drives HL15 case (link)
  • SFX-L PSU 600-750W
  • Asrock Rack embedded Epyc mainboard (link)
  • RAM to taste (cheap-ish DDR4 ECC)

Buy the basic HL15 case as adding a PSU is too expensive and too much power for your needs. Plus ample extra bays for future expansion of storage requirements.

The Asrock Rack board already has dual 10Gb/s NIC’s as well as up to 14 SATA ports. Added bonus is IPMI/BMC for remote management of the NAS.

Downside for both: not cheap! The case is 900USD, add shipping and you’re well above 1k USD. The board is not easy to find and I’ve seen prices of 2k USD. Maybe that’s come down now since EPYC 9000 is around, but I don’t know.

1 Like
4

If you want to run Ceph for HA storage, do NOT use consumer NVMe. They are horribly slow. Ceph wants drives with PLP, thus Enterprise/Data center U.2/3

I was considering MS-01 too for my stuff…but lack of space for multiple U.3 drives (and their cooling) made me consider other options. Mini PC is great value otherwise…perfect small and cheap node. And the 4x NIC on the MS-01 is great for clustering

Single server will always be cheaper and faster though. Unless you physically can’t put more hardware in a single server that is :slight_smile:

1 Like
5

Size, time, and interest. Been building systems since the late 90s, so I’m not averse, but I’ve done my time, so to speak. It’s no longer of any interest; I just want to buy the thing and send the thing away if it breaks.

HL8 is preferred to the HL15 due to size constraints. It’s also a pyschological constraint, because if I start buying big boxes, I’ll want all the boxes to be big, and then comes the rack.

2 Likes
6

This is a good advice; I was unaware of that limitation. Is there a meet-in-the-middle solution? I won’t need a lot of storage on the cluster itself, but I also don’t want it lingering in the back of my mind either, waiting for the wall to hit.

7

It is not a hard requirement, but Ceph relies on syncing all blocks written as a design principle. For Queue depth of 1 and 4k blocks, the difference is 100-200 IOPS (consumer drive) or 1000-2000 IOPS (SSD with PLP) per thread and SSD, so that’s a 10x difference. 100 IOPS is around the random IO performance of a local HDD. It’s horrible.

If all you do is sequential writes and reads…no one cares about IOPS. But if you don’t, you’re back to basically HDD performance.

Ceph runs on commodity hardware, but the drives need special (expensive) attention. commodity HDDs are fine though…but the WAL disk should have PLP too.

There were Enterprise M.2 variants around…not sure if they are still sold as I havent seen them offered in newer generations. E1.S basically replaced all M.2 SKUs while U.2/3 form factor is still going strong.

edit: if you are fine with SATA Enterprise SSD, these are still way faster than consumer NVMe cause PLP, they obviously don’t have the bandwidth, being SATA.

1 Like
8

That might be, in this case, putting it mildly.

Appreciate all the info; I don’t mind eagle-eyng ServerPartDeals or springing for a Kioxia u.2 if it’s the right move. I’d rather find the scope creep and extra costs on the front-end, then deal with the frustration after the build is done.

9

Doing some digging, I found some reasonably priced U.2s with PLP, specifically the Western Digital DC SN630 and SN640s.

I’ve made the decision to go with the MS-01s and to commit to using U.2 with PLP; I typically use Server Part Deals for purchasing drives, but I’m not sure about fitment and have come across the SN630s and SN640s being run in the MS-01.

Going to start with a single MS-01 and run one for a few months. Given the additional cost of the drives this feels like the right move; prove the model, and then build the cluster once I’m confident with the setup.

Any other specific drive recommendations that come to mind, perhaps specific models that offer good price to performance?

10

How much storage and at what price points?

My main recommendation the last year or two, Asustor Flashstor 12 bay m.2 drives (~$200 per 4TB plus base unit $1200) if you do not need more than 48 TB of storage and then everything else on a SSF Ryzen 9900X. Total cost ~5k but well worthcit for a home lab that consume 50-100W

1 Like
11

Enterprise U.2/3 drives like the Kioxia need very active cooling that the MS01 doesn’t provide (I think). Expect heat related issues.

IMHO these two quotes don’t go together. Maybe I misunderstood.

In my mind, building a CEPH cluster and maintaining that, while stimulating and interesting for a homelab for sure, does not fall into the “buy the thing and send the thing away if it breaks” category.

2 Likes
12

I have no interest in building the box myself (outside of installing the disks), but don’t mind spending time tentpoling and maintaining the box. The former is of little interest to me nowadays, while the latter is adjacent to my current interests and existing workloads.

1 Like