Ugh! Bruh… If you care about your data, you need to be more concerned about backups before all else.
Seriously, though. If I were the OP, ZFS would be the very first thing I would get rid of.
Reasoning:
A.) ZFS is a resource hog. It was designed to only offer performance benefits on capable hardware. As a home user, you’ll only observe performance benefits from running ZFS if your network speed exceeds the capability of your storage array. The OP is limited to 1 GbE. As you so kindly explained, a single spinning HDD is capable of 250% performance compared to his network. That means he’s already saturating what his network will allow. Which is why ZFS ARC adding a RAM cache isn’t helping his performance. He’d require a network capable of providing at least double his storage performance before he’ll see any tangible benefit. That requires upgrading all the NICs in all his computers and switches, which his low budget simply won’t permit. Also, it would consume more power. And last, why does he need it?
B.) I’ve said it before, and I’ll say it again. Snapshots are NOT backups. You still NEED a dedicated backup and recovery solution. He probably doesn’t NEED snapshots. Even if he does, then there are still better solutions to run considering his hardware constraints. For example, LVM also supports snapshots, but they are limited to volume level. If he simply must have filesystem-level snapshots, he could use BTRFS, which is less demanding. Since he only has 2 HDDs, he doesn’t need to worry about parity RAID, which requires 3, at least. However, I doubt he needs even that. Snapshots only provide resiliency, convenience, and continuous uptime. Unless you require being able to recover your server without having to take it offline, and I have my doubts, then this is merely a “nice to have”. Just remember to offset those benefits against the drawbacks.
C.) Filesystem compression doesn’t help files which can’t be compressed, like MP3, FLAC, MP4, AV1, etc which are already compressed as it is. You really need to understand what other file types you’ll be storing, and whether compressing them is going to save that much space. Also, remember that his system is bottlenecked by 1 GbE, and which his storage capability already exceeds, and so compression probably won’t help his performance, either.
D.) If he’s planning on running additional services, which he is, then ZFS is only going to starve his system of critical shared resources which could be put to better use elsewhere, like running all those container workloads. Without ZFS, he could switch to Linux which offers better support for containers, and which I would recommend over a hypervisor, because they can share system resources, without having to exclusively passthrough devices. That means he can run more workloads on less hardware.
E.) He can use the money he’d save being able to buy a less powerful system on some extra storage drives to install into his old server, which he could then use for dedicated backup duty. He doesn’t need to keep it running 24x7, since it’s only going to be used for occasional backup workloads. Just turn it on until it’s finished, and then shut down.
If he expects to retain ZFS, then he should seriously start to consider upgrading his network before all else. Even moreso than his budget constraits, that’d be his main limiting factor. Also, if I were him, I’d consider running Unraid over FreeBSD or TrueNAS anyday. In fact, the rumor on the street says that Unraid is the reason IX Systems decided to create TrueNAS Scale. They saw the writing on the wall and realized they’d have to come up with something which could compete with that. I think Scale is great… For those with the budget to permit it. For everyone else, there are even better options.
