RAIDZ vs. multiple mirrors is a trade off
- mirror VDEVS are faster
- adding mirror VDEVs you can do with 2 drives at a time
- RAIDZ VDEVs cost you less capacity once you go beyond 4 drives (but going beyond 4 drives means a bigger enclosure, etc. which adds cost and physical size).
- RAIDZ VDEVs can be slightly more resilient in that it doesn’t matter WHICH drives fail. e.g. if you have multiple 2 drive mirrors, you’re screwed if 2 drives fail in the same 2 drive mirror. WIth a RAIDZ2, you can have ANY 2 drives fail and still be OK. But you pay the penalty in terms of performance.
For a ZFS pool, your IOPs are dependent on the number of VDEVs you have. With mirrors, you can have more VDEVs with less drives, which means more (random IO) performance for the same number of drives.
For a single/small number of users? I’ll say “no”. I don’t have l2arc or slog on my ZFS nas and i’ve been happy with the 4 drive dual mirrors it has for the past 8 years, performance wise. It’s normally limited (or close to limited) by the gig ethernet connection it has.
All of your listed access patterns are fairly sequential/data archival type with a low number of users. Same sort of stuff I do on my home 2x mirror setup.
Also… i have a 6 drive RAIDZ2 at work for my personal workstation’s NFS based KVM data store. Performance running VMs on that is adequate for lab use over gig ethernet. Much more intense workload than you’re talking about… no l2arc or ZIL there either. That’s 6x SAS in RAID6 with 16 GB of RAM running FreeNAS with no tuning - over 1 gig ethernet to my workstation.
I would suggest that any SSD based cache beyond the memory cache is probably not worth it in your case and will just add additional complexity. If you had tens or hundreds of users (or much more intense data access workloads) it would be different. But you don’t.
What I’d do… use the ZFS pool with rust only, rely on RAM cache, and use the SSD as the boot/main drive for your system. Get a terabyte or two of SSD and use that for OS, games, short term current projects storage etc. Stuff you can afford to lose.
Back the “current projects” up to the ZFS pool daily/weekly/whatever.
In your situation i would (and have) get 4 drives (whatever the current price/capacity sweet spot is - which you did), set them up as dual mirror VDEVs, and one or more 1-2 TB SSDs for your OS/current data drive in your main box.
If you are concerned about the “2 drives failing in the same mirror” concern… then go RAIDZ2 or RAIDZ3, but to make it “worth it” IMHO you need to go to 6+ drives which as above means more initial outlay, more power, larger enclosure/more drive bays required, etc.
In that line of thinking my “real important” stuff is backed up to more places than my NAS. If your super important stuff is in nextcloud (for example) and kept on device, it will survive a NAS failure as it will be synced to multiple locations.
So for me, i consider 2x mirrors to be “good enough” for my purposes.
Thinking about this stuff before it happens is important. The other people who don’t are the crazy ones