My first NAS server build (Epyc Genoa, Supermicro 2u)

Hardware Hub Build Logs
, ,
1

After years of storing data wherever I had space and almost never making backups, I finally decided to get serious about my data security. This is my first proper NAS/Server build (ignoring the Pi serving my Home Assistant and a N100 based OpenWRT router), and the goal is to create a centralized, reliable, and reasonably secure home for all my data going forward.

All of this started when I scored a super cheap, old SC219-based server on eBay. With this chassis in mind, I put together a build using more modern components. I knew I wanted ZFS for the storage layer, enough PCIe lanes to handle both fast SSDs and future expansion, and a very fast network connection so I won’t be tempted to store data on my PC.

After some research and component hunting, I ended up with an AMD EPYC 9334 QS in a Supermicro H13SSL-NT motherboard. For RAM, I got a ridiculously cheap 8x 16GB DDR5 RDIMM kit from eBay. It’s more CPU and RAM than I strictly need for now, but it gives me room to grow.

Storage-wise, I’m running five Kioxia PM5 SAS SSDs in a RAIDZ1, with a sixth RM5 drive as a hot spare. Everything connects through a NVMe-capable SAS3 backplane to an LSI 9400 HBA. I plan on adding around 10TB (excluding parity data) of PCIe 4.0 U.2 drives once I get some for cheap. For the boot pool, I’ve got two mirrored SN850X NVMe SSDs. There’s also an external SAS HBA for tape backup, and potentially a JBOD if I ever need it.

Speaking about tape backup: I got a dirt cheap IBM TS3200 from, who would have guessed, eBay, which will handle my long-term backups. I’m still looking for an affordable LTO6 drive, but once that’s sorted out, it’ll be set up for automated backups using a quasi air-gapped media. Additionally, the tapes are cheap enough to store with family, just in case.

System Specs:

  • CPU: AMD EPYC 9334 QS
  • RAM: 8x 16GB DDR5 ECC RDIMM
  • Motherboard: Supermicro H13SSL-NT
  • Boot SSDs: 2x SN850X 2TB (mirrored)
  • Storage Controllers: LSI 9305-16e, LSI 9400-16i
  • Backplane: BPN-SAS3-213A-N8 (8x SAS3 + 8x U.2)
  • Storage SAS SSDs: 5x Kioxia KPM5XRUG3T84 + 1x Kioxia KRM5XVUG3T84 (hot spare)
  • Network: NVIDIA Mellanox ConnectX-6 + Broadcom BCM57416 10GBase-T
  • Case: Supermicro SC219
  • PSU: 2x PWS-920P-SQ

I’m still missing the rear window for the case as the server came with a proprietary WIO one. I’ve ordered it in March but Supermicro hasn’t delivered it, yet. The core system is up and running, and I’m working on dealing with some of the issues I’ve found (like this).

Here are a few photos from the build process:

Motherboard with the CPU installed

bld2
bld21920Ă—1440 448 KB

CPU with PTM7950

bld3
bld32048Ă—1536 444 KB

Cooler mounted, RAM and boot SSDs installed

bld5
bld51920Ă—1440 428 KB

Motherboard installed in the case

bld6
bld61536Ă—2048 477 KB

Temporary air shroud (will be replaced with this, soon)

bld7
bld71536Ă—2048 338 KB

Closed case with SSDs visible

bld8
bld82048Ă—1536 261 KB

If you’re working on something similar or have thoughts on ZFS + tape workflows, I’d love to hear how you are approaching it.

1 Like
2

Quick reply for keeping tap on your build. I am indeed working on something similar but actually fairly different. My main aim is redundant storage so after watching the NetApp video’s by LTT some time ago I decided to go High Availability for my NAS, meaning I now have 3 EPYC servers (2x 7551P based and a 3151 system) and I already got me a pair of 12-bay disk cages. And 3x NetApp DS2246 I got for cheap (with caddies, no drives). I need to figure out a cooling solution for the drive cages as the 6 drives now residing there get too hot. For financial reasons (electricity is expensive here!) the systems are offline. I’ve settled on 16TB drives and plain Linux mdadm RAID6 with JFS, but once BTRFS gets to play nice with RAID6 I’d be interested in moving over.

3

High availability is an interesting topic (I’d love to tinker with a system like that) but total overkill for my home use. As you already mentioned electricity isn’t free and cooling can also get quite complicated once you cross a certain power consumption. At the moment my system has a idle power consumption of about 160W which isn’t great but also not terrible. I’ll certainly try to lower it (maybe by adjusting the TDP parameter or something) once I’ve worked out all the kinks.

4

What else are you all planning on running? Thats a pretty sizable amount of compute for a NAS. I think my truenas VMs have 6 (maybe 8 max) cores available to them and neither come anywhere near utilizing all of them.

5

Besides some kind of backup software (probably Bacula) nothing at the moment. But this can change quickly…

1 Like
6

Like I said, main aim is redundant storage. I don’t need VM’s for extra services, no home automation here (the house is 100+ yrs old, not much to automate here) but I wanted EPYC+IPMI and both H11SSL boards, CPU’s and memory were reasonably cheap (bought just at the right time). Overkill, certainly. For now anyway, but who knows what the future demands :wink:

Having said that, I am considering putting a local cloud instance on the HA system, if only to be independent from the likes of google and amazon. But that’s a future project, if ever.

1 Like
7

You’ll have plenty of room to grow, which is a great thing to have.

Love the Epic cluster, but be aware that it is in no way power efficient and will be expensive to run, hot, and loud. If you’re really only looking for a NAS then this is a fair bit of overkill. You may want to consider a respec, or what you’ll use the extra compute for (since it will be drawing a bunch of power at idle anyway).

8

Tell me about it! :money_with_wings: It’s the reason everything is offline for the time being. My original NAS used an AM2 system and I still have it so I may swap out one of the EPYC7551P systems for the 3151 SoC board I have. Thing is, I don’t think I have a suitable internal HBA for it to allow all 6 HDD’s in the RAID6 to connect.

Just occurred to me: I also have a X470D4U system (with R7 1700 non-X) that has more PCIe slots. :person_facepalming:

I might be a while :wink:

9

Not necessarily. Sure, it all depends on the machines…old Broadwell Xeons or even EPYC Naples are orders of magnitude apart from how things are today.
EPYC Siena (8000 series) has amazing power-efficiency and perf/watt. Some guy here on L1T reported on 40W idle with basics+2 drives. That’s Ryzen/EPYC 4004 level of low-power, noise and heat.

The real deal on power comes with the drives you run. Because storage-focused stuff traditionally don’t need a lot of CPU horsepower and are low-spec.

AMD diversified the EPYC product range quite a bit in the recent years. Performance/socket isn’t the only metric anymore.

Cluster always uses more power. That’s just reality by having redundant components and running 3+ machines instead of one (bigger, but not 3x bigger on most stuff) machine with e.g. 3xVMs, single servers are also way more efficient. But you don’t build or run a cluster for efficiency or performance/€ reasons.

10

My experience with running first and second gen Epyc (which is what two of the machines referenced are) is that they idle really damn high. Not a bad thing if you’re hitting them hard as they seem to get more efficient the more load they get.

Perhaps “efficient” is the wrong term here. Let me rephrase to say that if you’re just running a NAS and a backup solution you can use far less compute that will have far less of a power draw, thus costing you less money.

I see that “efficiency” is a term thrown around about cost to run, when in reality is is really about how much loss there is between input and output.

To sum up my point. You can run a NAS and Backups with a 10th of the idle power that those Epyc chips will draw if you’re thoughtful in planning the hardware.

Think of it in terms of using a Ferrari to go get groceries on suburban American streets. Will it do the task, sure. Are there more cost efficient ways to do it in terms of vehicle and gas usage, also yes.

***edited to add analogy

11

if the input is power and the output is some performance metric/goal, it fits nicely. And is done all the time, be it perf/W, IOPS/W, TB/W, Tokens/W or whatever you are looking for.

If I can do the same stuff with half the power, it is twice as efficient. That’s just basic math.

12

Do we at least agree that a NAS can use way less compute and idle power than 160watts?

If so, neat!

Semantics aside, this is my point.

***edited for clarity

13

Absolutely! However given my original goals most low power options just wouldn’t work. The biggest limitation I had to solve was the number of available PCIe lanes. I needed 2x 8lanes for the HBAs, 16lanes for the nic, for the boot SSD I needed at least 4 lanes (preferably 8) and for the Backplane I need 8x 4 lanes. In total that makes over 68 lanes needed. That pretty much only left Intel’s or AMDs server platforms.

I will try to reduce power further once I have all the issues I’mencountering figured out.

14

I figured your Epyc choice was due to pcie lanes. I’m thinking that once you get it all up and running and find where the bottlenecks are in the rest of your network/systems you’ll be able to design something that uses less lanes and less compute if you want. But with the cost of all that gear for a NAS I’m guessing power isn’t a concern. What are you’re storage use cases? Are you editing larger video files from the NAS? Big CAD projects? Just wondering what the need for speed is about (if it’s purely fun or testing for production I totally get it).

My power considerations were more for Dutch_Master as he did state power was expensive and he was concerned about cost. An Epyc cluster will never be cheap to run. Without knowing the requirements for clients here I’m also just guessing however. Could be needed for sure though.

15

impatience :smile:. I do have a mix of data, and ideally I’ll be able to move my complete home folder to the NAS without any performance impact.

At this power consumption level, the NAS will cost me about 1,50 Euro a day which is OK (I’m monitoring it with a Shelly plug at the wall).

16

which will never happen. Network latency will eat most IOPS. Network storage will always be way slower than local storage.

1 Like