It’s not exactly that ZFS is worse at receiving damage after unclean power outage, it’s more that other file systems have a built in toolset to recover, often times automatically like with NTFS or XFS’s self-healing ability while the comparable hypothetical event (spacemap corruption) on ZFS is a much larger ordeal to remedy and can involve extensive downtime.
But this doesn’t show how fragmented the actual data on your pool is, only how fragmented the freespace is.
I think I’d find it helpful if we could articulate a few clear situations where ZFS wins by a lot, and how.
For instance:
I think C is obvious, but A and B?
Some may count a 1-2 drive NAS as not a real NAS but with single drives holding >16TB it’s still a wild amount of storage.
(That said in our current financial climate I find myself with no large drives and a random assortment of mismatched borderline ewaste, possibly bcachefs looks the most interesting there)
My current understanding is that for A, you at least have ZFS warning you if data has corrupted, plus snapshots. So I suppose you would know to restore from some other backup, because there’s no self healing. For B, a mirror gets you self healing and snapshots, potentially a boost in read performance.
Snapshots seem like the main thing that makes zfs worth considering even on single drives. but…surely other file systems have something to offer there though?
Fair enough. But if you just write data once, it can’t become fragmented. Fragmentation happens when you write, delete, rewrite, etc. Even edits aren’t too bad because of the copy on write method, as long as you follow the guidance on not overfilling the pool.
During a scrub and resilver it reads all the way through at over 800MB/s. It wouldn’t do that if heavily fragmented.
I wouldn’t use ZFS on a single drive configuration. The CoW methodology has more performance downsides than traditional file systems, and offers little benefit (at least that you can’t accomplish in other ways)
As for a 2 drive mirror, ZFS is great. I use it in that configuration all the time. Great configuration for random performance. RAIDz, RAIDz2 and RAIDz3 are great too for their given applications.
Reasons for this include some of the best data corruption protection possible, block based snapshots that are easy to manage. Incremental block based send/recv actions.
btrfs can achieve similar benefits in theory, but I have heard of nothing but problems from users of btrfs
That is all based on a IMHO false premise.
Your SSD from 2015 is some generic brand like Adata that slapped on some Phison controller with a firmware bug that lies about sync writes? And you use two of them in a mirror instead of mixing vendors? Than yes, what you describe can happen.
You have issues with your RAM but don’t have ECC? Yes what you describe can happen. But unlike NTFS, you would at least notice it with ZFS.
I mean, it depends. On my OPNsense firewall with some shitty 128GB SSD I had lying around, course I would use ZFS and not ext4.
I probably would not use it on my gaming PC.
This is a general problem of this thread, we are just exchaning hyptheticals, because @Moog has not shared what his/her actual use case is. Because again:
Btrfs has a similar feature set to zfs (better in some areas as it can also defragment and its offline filesystem check and fix tool does work well enough), but does not support raid 5 or raid 6 setups.
That is fair, but that is a rather specialized use case where performance doesn’t matter much, and the benefit of snapshots outweighs it, but quite frankly, just to save myself the headaches if something corrupts, I’d use a ZFS mirror in that case.
In my particular OPNSense implementation it runs as a VM with passed through PCIe NIC’s. The host stores all of its VM drives as zvol’s on a redundant ZFS pool. Because of this, it seemed more efficient to just install OPNSense with the default old school UFS unix file system, and handle snapshots on the host rather than nesting ZFS instances. But this is also a rather specific use case.
Yep, that is true. My assumption based on the little data we have is that he is setting up a basic file server/NAS, so that is what I have been using in my recommendations
I asked a broad question because I got frustrated looking for recommendations and seeing “zfs is best for data integrity” and couldn’t help but thinking “isn’t that the whole effin point of this whole exercise?” And figured I would try coming at it from the other side.
as for my actual use case, I’m building my first home server which is primarily going to be a NAS with the intention to expand into self hosted applications like jellyfin after.
While this discussion has been ongoing I’ve loaded 3x 14TB drives, 2x 960GB SSDs, and a 1TB ssd into a server running Unraid. I put the 3 HDDs into a RAIDz1 pool, the 960 drives into a zfs mirror pool, and the 1TB is a standalone ZFS pool. I have purchased an HBA to add a further assortment of HDDs of differing sizes I have laying around. I’m not convinced the single device pool is providing much benefit, but having snapshots at least seems like a decent idea.
Maybe I’ll try TrueNAS by importing the zfs pools there, but I’ll wait until I have cloud backups sorted before attempting to change os
ZFS is “so good” for enterprise setups that run 24/7, are not terribly concerned about power draw, and they’re buying all the drives at once and they’re all the same size. It supports fancy stuff like the special vdev devices Wendell has gone over in some videos/threads here too.
If you’re homelabbing and don’t mind spinning your drives always, your arrays are made of equally sized drives, and you set it all up at once, ZFS is real nice, and TrueNAS is the easiest way to do so for a NAS, yes.
If any of these conditions aren’t met, then ZFS probably isn’t the best choice, and that’s why consumer NASes usually don’t use ZFS by default.
The reason they dont use zfs is mostly because they dont care, not because its a bad choice.
Most consumer nas appliances (excluding qnap that uses zfs) are using mdraid (aka standard linux software raid) which has the same base limitations of zfs but none of the benefits (besides being a lot lighter on cpu).
All drives must be the same size and all must run at the same time when data is accessed.
Sinology does a bit of shenanigans layering mdraid or ldadm and creating a btrfs volume on top to have compression and snapshots ( with an ancient btrfs version at this point) so they can have a more flexible raid setup with Synology Hybrid Raid.
But thats it, afaik nobody implements a system like unraid (its actual native parity raid implementation) or snapraid+mergerfs that allows most drives to sleep.
They just do the bare minimum to claim they support some feature, slap a cool web interface on top, and done. Letting disks sleep piecemeal is not expected on raid, using mismatched drives is not expected on raid. So mdraid is good enough, send it.
PowerEdge XR2’s were where I saw this happened most often, so plenty or RAS features including ECC were present… however we were using SATA drives so we didn’t have FUA available which maybe could have made ZFS more reliable? Pretty sure the drives were PLP equipped HK4Es so I’m skeptical it was the drive letting the system down reliability-wise.
After several months of problems with ZFS we switched to ext4 and stopped having issues, despite an uptick in power excursions.
My personal experience is with the little QNAP box at a family member’s home business. They offer the option to sleep the drives (since it’s a mirror array, I assume it’d sleep/wake both at the same time). At work I use TrueNAS, I’ve never deployed a prebuilt machine.
Methinks given the way many home NAS boxes are used, sleeping the drives is probably fine, as they aren’t accessed super often, only when shuffling files around. Usually folks use them as a storage box for movies and family photos and the like. The above family member does use their NAS as a network share for work files between multiple computers, so they probably would benefit from ZFS. At the same time, the current setup hasn’t failed or lost data, so 
.
They’re often really low-power chips that barely run said webUI, so the CPU needs of ZFS would probably be an issue for more users than the lack of ZFS features is.
I think if a raid solution or drive format does what you want, then it’s good. I use XFS for my home drive because I run Gentoo and that’s what they advise, it’s very fast and deals with compiling workloads well. My storage drives are in a BTRFS mirror, because ZFS lags behind kernel versions, and I usually run the latest stable (and often release candidates as well). mdadm is annoying to set up, BTRFS isn’t, so BTRFS won. If I eventually get home internet again, I’ll probably put those drives back into a ZFS array on a dedicated machine. Not that I need the features really, it’s just nice.
Ah, so if we want to detect bitrot (and sometimes repair) in the underlying storage we could leverage one of:
I think in the user-space backup tools are worth a mention, if we are wondering about ZFS and data integrity it’s because we value our data, ergo we ought to have backups!
In my case I use restic, and routinely have the destination validate it’s own integrity, then I can be happy my backups aren’t bitrotting regardless of the filesystem they sit on.
I can then validate the files sitting on the backup source against the backup destination, by telling restic to ignore mtime etc and reread the files.
This all sounds a little manual, but I find it easy, each of the two operations is a one liner in a shell script.
Unfortunately whatever you do, it needs some amount of setup and human attention. It’s not unheard of to lose a mirror completely due to dying disks, because the first death wasn’t noticed. Ditto in backups, it’s entirely unsurprising when someone who setup their backups once and never looked again, finds that they can’t restore when they need to.
Edit: I see backups and NAS as intertwined topics, that influence how I use drives. Like given your 3 14TB drives instead of RAIDz1, I’d mirror 2 in the NAS and put the third elsewhere and backup to it.
OPNsense is an appliance like TrueNAS. You can save ALL settings by downloading a simple XML file (or back that up to a cloud drive). Using a mirror would be a waste in an non critical environment. I just reinstall it and import my 500kb xml file 
Course. Otherwise you would have ZFS on top of ZFS or CoW on top of CoW. I also only use ext4 for my VMs that are stored on ZFS. I would not virtualize my OPNsense, but that is another topic 
I mean nothing really has changed, so IMHO the only thing that has actually changed is that you no longer notice it, because ext4 won’t notice it.
I do. They are great, but pretty expensive.
That is a hard one. So I will throw some bulletpoints at you.
I used to use the predecessor to TrueNAS named FreeNAS back in the day, but as I learned more I ditched it in favor of running ZFS directly on the host utilizing a combination of NFS and SMB configured in Linux. It does everything I need without the feature bloat of TrueNAS.
Back in the FreeNAS days, it was FreeBSD.
Nowadays, it is just a Debian with good defaults and a nice GUI for NAS management.
I don’t really see much bloat to be honest, but that probably depends on your definition on bloat.
Yeah, back in the day I ran FreeNAS as a VM on ESXi with a couple of passed through SAS HBA’s for the drives.
But by the time I was switching to Proxmox in 2016 I decided it was a waste of resources and added complexity to run FreeNAS insode of a VM and pass it through, when Proxmox itself (due to being Debian under the hood) had native ZFS support, so I just imported my pools straight into Proxmox.
If I had a standalone NAS box I’d probably still be running FreeNAS/TrueNAS just for the simplicity of the defaults and management, but since I was already running Proxmox I just didn’t feel it made sense.
I have set up NFS shares for my other servers and static Linux desktops directly in Proxmox (command line, Proxmox does not have file sharing management in the GUI) and I created a small dedicated LXC container to handle sharing SMB to other clients.
All of the VM’s and containers on that main Proxmox node that need to access the NAS do so natively directly without any need for network shares.
It is a system that has been working quite well for me for some time now, but I could do it because I slowly built up familiarity with managing ZFS pools from the command line when I was troubleshooting FreeNAS over the years, and that is what made that transition easier for me.
If I were using SSD’s in my NAS i would probably rethink using zfs. but I don’t-my NAS is all rust, all the time. and for that, as a dedicated storage only device with plenty of memory, it works great. Running an 8 wide Vdev with two hot spares and ECC memory has turned my NAS into a “set it and forget it” set up which is what matters most to me. I do not want to spend my time configuring the storage server, I know some people do and that’s fine, i just want it to work. ZFS does that well.
I feel the same way about networking, I was running a OPNsense router and regularly managing the upkeep and maintenance-- i don’t wanna do that- I want to play with the LLM server- so I moved my network to the Omada system and now I don’t spend but 5 minutes checking up on it and verifying that it is doing the business, which is all I want- I want it to work