Hello friends!
Unfortunately, due diligence hasn’t really paid off this time - Google gives me a lot of TrueNAS forum posts from 2013, and a lot of Reddit posts where the OP and commenters have deleted their accounts and messages, so I’m having a hard time gathering any useful information on this one. Instead, I’ll ask you guys (I tried, promise!)
I’ve built myself a new NAS. Right now, it has eight 16TB Exos drives i there attached to an HBA. Two 250GB NVMe drives are acting as a SLOG (I tried to do the overprovisioning trick on them but got errors about namespaces, so I’m hoping the wear levelling does its job in the meantime), and two mirrored 500GB SSDs on a carrier card acting as a special vdev.
The drives are currently configured in 2x4 RAIDZ1. This seems relatively reasonable, however, I’m considering scrapping the array and rebuilding it as RAIDZ2 before there’s enough stuff on there that can’t be moved.
Considerations:
- NAS has been set with a max ARC size of 100GB, since there’s 128GB installed.
- It’s on a 10Gbit network. The two vdevs don’t seem to saturate it as is, so that’s not a huge concern.
- I intend to use this for VM backups and uni work. The uni work in particular is something I’d like to be safe.
- The NAS will run a small number of VMs - I’d like to try out Grafana Mimir and learn k8s, so the VMs will be split across the dedicate VM host and the NAS for some basic HA.
- I do have a few sites that are lightly used. Database is Postgres in a VM.
- My VM server is using some very old SSDs right now. I may transfer these to the NAS and run them over NFS - in this case, RAIDZ1 has better IOPS, but I’m likely to be the only real user of them. I do have a couple of 1TB NVMe drives I can put into the VM server instead, if I don’t mind kneecapping them to only run at x2 speeds. These were originally intended as cache drives for the pool, but a shortage of PCIe lanes means my carrier card can only run with two unless I move the drives to SATA instead of an HBA.
- Uni work will involve VM usage at some point, and I’ll likely need to be using NFS for portions of it as a mounted file system.
- The NAS is running TrueNAS scale, and I have a few apps running, including Prometheus. I also have Minio set up as a backend for a future Gitlab install.
- There is some data on there that I got through torrents a few years ago, for which the torrents seem to be dead now and that I can’t find a replacement. Otherwise, I’ll be mirroring a bunch of ROMs and ISOs.
- I’m running a Storj node off it and farming Chia to make use of the spare capacity that I’m not using yet, so a pool failure has a financial penalty, sort of?
In either case, the capacity works out the same (I’m excluding mirrors as an option here, purely because I want the capacity - I’d prefer not to have to upgrade in that regard for at least 5 years, which is when the warranty on the drives expires anyway).
I’m vaguely aware that sequential throughput will be more or less the same in either case, but I’ve found conflicting reports on that, so I’m not entirely sure? Random IO is less important, especially with the special vdev handling metadata and small blocks, but it’d be nice to find a decent compromise.
With this in mind, I’m leaning slightly towards RAIDZ2 and eating the penalty on random IO - I think the user count for the projects I run is low enough for the time being that if I run the database over NFS, SLOG and ARC will save me until I can get NVMe installed in the VM host proper. However, I’m relatively new to ZFS (in the sense that my prior experience over the last few years has been “oooh, shiny” and just doing what TrueNAS told me with what I had lying around), so I’d definitely appreciate some advice for my situation from more seasoned users.
Thanks in advance!