I’m trying to figure out how many disks I need for my TrueNas build. All I know so far is it’s going to be ZRaid3, and I have five 24tb disks for it. I might have to up it to eleven to maximize disk size and reliability. I’m going to use a SuperMicro Sata Dom mirrored and stripped for the boot drive.
Currently, I do not have any cache SSDs, nor spares planned. Depending on if my backup and media workload would benefit from any of the cache types. I might need space for additional swappable disks. NVME would be an option, but I might save money by going with the 2.5 form factor.
I still have to figure out the server cpu and motherboard. I should also add probably 3 spares. Not really sure how many I should have. So, that’s about 14 disks there. I should also have a few nvme and sata slots on the motherboard for gumstick drives.
I’m trying to minimize noise, and am trying to decide between a Storinator 30 bay chassis or HL15. 30 bay I’d have options, but the 15 might be more silent. Think a 15 bay would be enough for leaving room to expand cache disks in the future?
Have you accounted for a boot SSD redundancy?
You want the boot pool to be redundant to keep all of the configs, and from how iXSystems build their systems, either have the SSDs on a separate PCIe carrier board or use the M.2 slots on the motherboard.
You do not want to have your boot media on the same backplane/HBA with the rest of your media
I’d advise against SATA DOMs unless you’re going to have some spares laying around or know how to get one very quick
You do not want to end up in a situation where one of them fails and your boot pool degrades, and have to wait for another one to arrive while running on a single DOM that might be failing soon
Why is a mirrored boot drive that important? My truenas stores most of the system files in my nvme drive pool. I back up the configuration and it can be easily restored after re-installation, it’s an 800kb tarball. I would not bother using up resources for a mirrored boot drives unless this is some high availability enterprisee setup. Homelabs do not need it.
I setup 6 mach2 exos dual actuator drives in with 2 raidz2 vdevs and ssd special vdev and performance is very respectable and almost saturates 10Gb link. I did a lot of fio testing and mirrored vdevs were definitely better but i wanted moar TBi and I have a separate fast nvme pool for IOPS.
For a homelab, yeah it’s probably not that important. Ideally, you can just wipe and reinstall the OS of your NAS any time you’d like, because all your data should be kept separate from the OS.
It doesn’t hurt to set up your homelab in the same way as you’d do things in production, but budgets are a thing which can get in the way of that.
+1 on this. We would all like to have a 100% uptime over 20 years with the same system if possible, but that gets really expensive really fast. Better to have 99.99% uptime for a fraction of that cost, especially for a server that does not have to be up all the time.
As for how many disks you need - It is mostly a question of how much you need to store. If more than 6 physical disks you should consider RAID6, else RAID5 (or rather, the ZFS equivalents).
However, also consider the bandwidth, here - a 24 TB drive has a pipe of up to 180 MB/s. That means a restoration of 20TB of data takes around 31 hours to restore. HDDs desperately need to get faster in transfer rate as they increase the capacity, but I believe that is just not happening. We are simply at the end of the technology and at 30TB-ish, the tech is simply too slow to make a difference. Not to mention there is a 256 TB SSD coming out in a year or two…
@redocbew hit the nail on the head for the reasoning.
In my opinion it’s best to set up your homelab the same way as you’d do things in production. When you have to deploy a system such like that, you can use your lab as a production deployment blueprint for work.
The satadom is so I don’t take up expansion real estate. I’m just mirroring it to avoid down time in case of catastrophe. Satadoms are fairly cheap too.
I would stay away from SATA DOM’s. They seems to have a high failure rate. I would use optane instead. You can find 16GB optanes for under $10. Which 16GB is plenty for TrueNas.
I don’t think it would make much sense spending so much money on disks just idling there unless you have the need for absolutely zero down time.
I’d keep them around in a cabinet and swap in as needed. That would let you save up on useless sleds in the server and give you more space for cache drives, scratch drives or space for a new pool.
For the price of one more drive might I suggest a striped Z2 array? Would be double the read performance and net you the same reliability.
If you’re comfortable with standard ATX cases and don’t need or already use a rack solution I’d suggest you skip those overpriced boxes and used the saved up money for a cold backup with a couple 24TB drives. Maybe even a compressed and encrypted one.
The disk situation is honestly up to your wallet if you already meet the minimum req for Zraid3.
Since you’re going for Zraid3 over Zraid2 I get the feeling you want reliability from nas. Personally I’d look into what the price difference of having a mirrored server offsite running zraid2 would cost since a single super reliable NAS isn’t a backup.
2 cheap tiny sata ssd’s for boot media if you want to, many just go usb but it’s generally 0 budget impact to go for the “pro” option for boot media.
Running redundant PSU and using a UPS is what I do for peace of mind.
CPU+motherboard is in my opinion anything that has the features you want and can run ECC memory. It’s not easy to figure out if something will work with ECC so it’s not a bad idea to see what others report as working.
In my experience cache is overkill if you get a decent chunk of ram. Ram+rust works really well in truenas if you don’t skimp on the ram.
If I remember correctly the rule of thumb is roughly 1GB of ram for each TB of usable storage. If you end up with 15 24TB drives then 128GB of ram might not be optimal but you can give it a long hard thing about how much storage you’ll actually end up using.
As for noise, it’s going to be audible unless you isolate it somehow.
Cache works to a point. It’s like the rule of thumb of a swap partition disks being 2x your RAM size. Was a great advice when RAM was small, but these days you are quite unlikely to run into problems if you have a 16GB Swap and 32 GB of memory.
You might, but, very unlikely. At the same time, 1GB swap partition is by far too little for that amount of RAM.
Nowadays, 64 GB of RAM is plenty for most drive pools. You are not going to saturate that unless you are doing a full backup of the entire pool.
Considering your ZRaid3 configuration with 5-11 disks, 3 spares, and plans for potential cache SSDs and expandability, the Storinator 30-bay chassis would be a better choice over the HL15. While the 15-bay might be quieter, it leaves little room for future growth. The 30-bay chassis offers ample space for your current setup and future expansions, allowing flexibility for implementing various storage solutions as your needs evolve. To address noise concerns, look into noise-dampening solutions like foam insulation, vibration-dampening mounts, or replacing stock fans with quieter alternatives. This larger chassis effectively future-proofs your setup, giving you the freedom to experiment with different configurations without physical constraints. The extra bays will prove invaluable as your storage needs grow, especially if you decide to add cache SSDs or expand your array in the future.
