I started down this path over a year ago and play with some stuff at work (a bunch of really old hardware just laying around). This is the design for my new home server to replace my X299 HEDT computer that currently just hosts PLEX. It could have a lot of capability but was my old gaming rig, then decided to build a NAS, but didn’t hear about ZFS till later. This entire process seems like a money pit (it is) but I’m learning more along the way, and well since I’m an IT in the Navy, retiring in a few years, the experience and knowledge could be useful.
TL:DR – Need help and clarification RAIDZ design/setup – there’s lots of data and a explanation.
This is a log/story of my learning experience and poor planning build, I have watched more YouTube videos from Lawrence Systems, Level1Techs, ServeTheHome, 45 Drives, Techno Tim, Raid OWL, and Craft Computing for ZFS. I’ve also scoured the net looking up information about what to do with my build and have almost completed my build. I felt the need to rush the building process a bit recently as I’m in the US Navy and moving overseas (few months) and will deploy the server to a family members home while away. I plan to use VPN to access/manage the machine, but the family member also works in IT, so I feel safe having it there. My current server has 42TB’s used and I have about 1TB free, I move files between externals currently to mitigate before the new server build.
Here’s a list of current parts and parts that are in shipping, then I’ll talk about the plan, and how the server is setup.
| Part | Name | Model # | Quantity |
|---|---|---|---|
| Chassis | 45 Drives Q30 | Q30 | 1 |
| CPU-cooler | Noctua NH-U9 TR4-SP3 | 1 | |
| Chassis-Fan | Noctua NF-F12 iPPC 3000 PWM | NF-F12 IPPC 3000 | 5 |
| CPU | AMD EPYC 7302P | 100-100000049WOF | 1 |
| Motherboard | SuperMicro H11SSL-NC | H11SSL-NC | 1 |
| RAM | Supermicro 64GB ECC RDIMM DDR4 3200 | M393A8G40AB2-CWE | 2 (+2) |
| PSU | Zippy 1200watt Dual Redundant PSU | 1 | |
| GPU | NVIDIA RTX 4000 (Quadro) | 1 | |
| HBA | LSI 9305-24 (SAS HBA) | SAS 9305-24 | 1 |
| HBA #2 | LSI 9305-16 ( not purchased ) | SAS 9305-16 | 1 |
| NIC | 10GigTek – Dual SFP+ | Intel X710-BM2 | 1 |
| Expander | SuperMicro PCIe carrier for 4xNVMe | AOC-SHG3-4M2P | 1 |
| NVMe Drive | Kingston DC1000B 240GB m.2 | SEDC1000BM8/240G | 2 |
| NVMe Drive | Micron 7400 PRO M.2 22110 1.92TB | MTFDKBG1T9TDZ-1AZ1ZABYY | 2 |
| HDD | Refurb – Seagate Exos 14TB X14 SAS3 | ST14000NM0288 | 15 (+6) |
| OS SSD | SAMSUNG PM893 2.5” 480Gb SATA3 | MZ-7L348000 | 2 (Mirror) |
As you can see lots of capability beyond my current needs, but most of the hardware is already purchased and installed. The +2 on the RAM I’m probably going to pick some up here shortly as it’s much cheaper now and instead of doing L2ARC for my purposes. The Seagate Exos drives are shipped, and I’ll be getting the build moving forward, I purchased 20 drives (reseller threw in an extra, woo (21 Drives), but only plan to use 15 in the Pool.
Server currently runs Proxmox VE 7, plan is to spin up a couple of VM’s Truenas Scale (ZFS), Ubuntu (Plex), Houston (Server management), and anything else early on to install (sonar, radar, other download management). I was thinking about using the Kingston NVMe Drives (Mirror) as a ZIL/SLOG location, and the Micron NVMe drives (Mirror) for either hosting VM’s or being a location for incoming downloads? (Not sure).
The ZFS layout I had planned was a Pool of 15 Drives (RAIDZ2) with two VDEV’s of 6 disks each. 45 Drives Q30 Layout, backplane is SAS interface.
| Group 1 – 15 Drive bays | Group 2 – 15 Drive bays |
|---|---|
| RAIDZ2 – VDEV1 | Future addition |
| RAIDZ2 – VDEV1 | Future addition |
| RAIDZ2 – VDEV1 | Future addition |
| RAIDZ2 – VDEV1 | Future addition |
| RAIDZ2 – VDEV1 | Future addition |
| RAIDZ2 – VDEV1 | Future addition |
| RAIDZ2 – VDEV2 | Future addition |
| RAIDZ2 – VDEV2 | Future addition |
| RAIDZ2 – VDEV2 | Future addition |
| RAIDZ2 – VDEV2 | Future addition |
| RAIDZ2 – VDEV2 | Future addition |
| RAIDZ2 – VDEV2 | Future addition |
| RAIDZ2 – Hot Spare | Future addition |
| RAIDZ2 – Hot Spare | Future addition |
| RAIDZ2 – Hot Spare | Future addition |
I don’t need personal opinions on how I could have spent way less to get the same result, that’s apparent, but would like to know if what I’m looking into doing is correct and not completely dumb in the setup. Below is what I could figure out at ( ZFS Capacity Calculator - WintelGuy.com ) RAIDZ calculator.
I do have a QNAP external with 56TB of space for backups, and will most likely upgrade its capacity in the future as I’m only using 6 of the 12 bays (Has dual 10gb SFP+ NIC).
Thanks for all input.
-LGG