Title. I keep reading that ZFS is the best for data integrity, has good performance, and has magical snapshot and replication functionality. That all seems rather fundamental to the point of having NAS to begin with, so I’ve been assuming that I need to use ZFS. And if you want to use ZFS, surely that means you ought to be using TrueNAS if it’s starting from scratch, right?
My doubts have been creeping in though. If ZFS is so good, why isn’t it used in NAS appliances like Synology? Are there things that other file systems do better? Is it worth considering if you can only afford 3 drives to form a single vdev?
Just more generally, if you do have the disk capacity and system resources to run ZFS well, when might a different file system still be the better choice?
I can think of 3 general scenarios that can be challenging with ZFS that should be considered before using it:
Lack of power stability: if you can’t grantee uninterrupted power to the machine running ZFS, consider a more resilient file system/volume manager that can more gracefully handle outages without corruption.
Using SSDs: ZFS takes away much of the benefits of using speedy SSDs by slowing down the speed of access to data on them.
Expecting data to last a long time on a pool without drawbacks: ZFS will fragment the data stored on it’s pools and has no “easy” way to fix (defragment). This leads to slower data transfers and increased mechanical wear on the HDDs.
Synology did not use it because they want more flexibility, as zfs is not as flexible as their own raid implementation where you can change the type of raid at any time and use mismatched drives.
What? Zfs is a COW filesystem. Why would it not handle power outages.
Zfs rewrite does that now.
If you need drives to spin down and dont care about high throughput, and most of the data is static like in a media server, you can also use snapraid.
Maybe? Its a storage appliance so it simplifies setup of a nas, but you can roll your own or use the zfs plugin for cockpit web interface too on a diy setup
Because free space map appendations are not journaled and also happen at unpredictable (and seemingly frequent) times making them especially vulnerable to power outages. Technically ZFS isn’t corrupting the user data it stores when it corrupts the freespacemap after a power outage, but the pool is probably not usable after the event.
Once the pool is aged there is no guarantee that rewrite will be able to rewrite the file(s) to continuous physical blocks. This is something a true defragmenter would do.
On NVME, XFS and ext4 beat the pants off ZFS in a lot of workloads, especially on random and latency.
ZFS sometimes breaks on new kernels, and there’s growing concern that more and more kernel interfaces will be made GPL-only in the near-ish future.
ZFS ain’t perfect, but those are the only two reasons I can think. It’s also kind of an indictment of the dev process for filesystems these days, that things either grindingly improve only on the scale of decades (btrfs) or crash and burn into obscurity (bcachefs). (Not saying I’d be any better at such a complicated thing, I get it, but such filesystems seem to be a pain point for FOSS.)
What about when you have just 3x 14TB drives plus a mixture of smaller drives? What is the smallest size that ZFS can still provide benefits over something like btrfs?
ZFS was primarily designed around the idea that storage is incredibly slow, and CPU/RAM is incredibly fast. Many, many trade-offs were made to take pressure off of the storage and place it onto the rest of the system, which is a little counterproductive in the era of NVMe drives that can casually handle a million IOPS and hundreds of simultaneous operations. It’s a product of the era. It was also designed primarily with data integrity in mind. Using ZFS I’ve lost data exactly once and that was a me problem, not a file system problem. Every piece of hardware in that box was on the edge of dying because I built it out of garbage. I know, I know, sample size of one. But trust me, it’s an interesting sample.
Every time I’ve used Storage Spaces/ReFS, I have lost data.
Every time I’ve used BTRFS, I’ve lost data.
Every time I’ve used EXFAT, I’ve lost data.
NTFS, EXT4, ZFS, and APFS have never decided to randomly eat my data.
NTFS and EXT4, however, won’t keep data indefinitely, as I have confirmed bit rot using both of them. APFS is still new, but probably still has a similar issue. That leaves one.
If BTRFS had been the oracle-free cleanroom rewrite of ZFS that everyone expected it to be, then I would suggest that. But unfortunately, they stopped at parity because it turns out that implementing a file system with working parity checking underneath is actually difficult. See the recent thread about someone vibe coding a fix for it with Claude and how far they’ve managed to get and not get at the same time; the TLDR, as Claude got most of the way into it, realized that architectural decisions that were made early on meant that it was more or less not possible to implement RAID Z1, Z2, etc.
As far as I’m concerned, the only file system you should be using if you’re planning to keep your data in perfect pristine condition for more than five years is ZFS. Full stop.
Not the log space map but the free space map, two separate constructs. When the free space map appendation becomes corrupted from an unexpected power outage, ZFS no longer knows where the freespace in a volume is and is supposed to enter readonly although often times the system will just kernel panic instead of giving more verbose clues.
Pools with lots of free space can also have the issue of rewrite not effectively defragmenting files if the pool has been aged by writes and then deletes causing a lack of continuous LBAs. This is a pretty typical scenario for a pool that is years old and has had some use.
Dammit i mixed names above. The log space map is the journal of the free space map in the actual metaslabs. So yes there is a journal for free space map.
From your explanation I think you are confusing this alleged lack of journal for free space map with the infamous metaslab free space corruption bug. Which afaik isnt clearly linked to unsafe shutdowns or lack of journal in any way, people reporting it say it happens randomly, sometimes even while the system is running. Since its been elusive to reproduce reliably nobody has worked much on it.
You’ve touched on what I think is one of the main arguments as to when ZFS probably shouldn’t be used.
Adding and removing storage to ZFS comes with some major caveats. Perhaps most notably being that when you add a vdev to a zpool, then almost without exception, the only way to remove it is to destroy the entire zpool. This also means that if you mix the redundancy / parity within a zpool, your zpool is only as resilient to total failure, as your least resilient vdev.
A major feature of ZFS is it’s data integrity features, some of which only make sense with some form of redundancy. Which means that adding individual disks usually doesn’t make sense.
When using redundancy, be it a ZFS Mirror or RAIDz1, 2, or 3, the total capacity of the vdev is limited to a multiple of it’s smallest disk. If you were to mirror one of those $100k+ 100TB SSDs, with a 120GB 2.5" SSD… congratulations, you’ve got 120GB of storage.
[EDIT] One other thing I just wanted to touch on. If you choose to use ZFS, then TrueNAS is a very good option for an OS but it is by no means the only game in town. I use Proxmox for example. It is a little more work, being a bit more DIY in terms of ZFS management and the various other services that TrueNAS exposes via it’s GUI but the learning curve for the basics isn’t that steep. Plus, you could always install a ZFS management frontend.
At the time I made the switch (back when FreeNAS was still Jails only), it allowed a lot more freedom in terms of running the services and VMs that I needed. I’m also at that point now where I’m limiting the breadth of tech that I need to learn about. I use Debian on my desktop. So anything I learn for the Debian based Proxmox, is usually applicable to my desktop. Which is also an argument against the version of TrueNAS based on *BSD but perhaps not for the Linux based released.
What gives you the impression that ZFS does worse than other solutions in a sudden loss of power situation? The ZFS Intent Log is supposed to be world class in this regard.
Agreed, though I’d probably limit the statement to NVMe. it can still take advantage of SATA and SAS SSD’s pretty well, but once you get into NVMe speeds it starts showing its performance limits.
NVMe SSD’s still do well in special purpose applications with ZFS (like for Special VDEV’s to speed up metadata/small files), LOG devices (dedicated Intent Logs drives to boost the speeds of sync writes) and L2ARC though, as these are not as CPU dependent as reading and writing from the main data devices.
In my experience the performance issue is really a NVMe main data pool thing.
If you write and rewrite data a lot, yeah.
Lots of NAS applications are practically speaking write once, read many type applications.
My main ZFS pool in my NAS has a create date of 2014. IN that time it has grown from a single VDEV of six 2TB consumer drives, to two VDEV’s of six 16TB drives each (and a mirrored SLOG, and L2ARC and a triple mirrored Special device)
Fragmentation is still pretty OK…
root@proxmox:~# zpool list zfshome
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
zfshome 176T 104T 72.0T - - 8% 59% 1.15x ONLINE -
It’s a cool feature, but ultimately it winds up being mostly useless for most people, since doing a zfs rewrite dereferences your snapshots.
On the rare occasion I’ve needed to do something like that, I just did a zfs send/recv to a different pool, and then sent it right back to the original one.
Power consumption is one reason. Fake RAIDs use a lot less energy.
Other reasons depend on your workload, your data and on your hardware.
RW amplification, CoW, padding, pool geometry, fragmentation CAN be an issue.
Emphasis on CAN.
For most it isn’t. And you can do many stupid things. Like changing ashift, block size or record size, without understanding how they work. But if you use the defaults, you are probably fine.
You’ve been lucky enough not to experience a power outage when the pool is updating the space map at some critical time; I have not been so lucky.
The issue with ZFS isn’t that it is reporting corruption issues, it’s that the pool becomes unusable after the spacemap appendation corruption and there is no easy way to recover. If you’re lucky the pool mounts read-only and you can copy the data off of it, if you less lucky you will just receive kernel panic.