I have a flash array for my nas, and it works fine with zfs.
The bottleneck is the gigabit network port, but the latency/snappiness is still nice to have, even though ZFS by design adds overhead, in order to preserve integrity.
I mean, they are only SATA SSD’s so not amazing, but still handy to shuffle incremental backups to and fro using sanoid’s syncoid for backing up other devices
My experience is that ZFS still does a pretty good job of keeping up with SATA SSD’s (as long as the system it is running on is decent and able to keep up).
If properly set up, it even does a good job with dual port SAS3 SSD’s in multipath mode using both ports at the same time. These drives read at about 1,900MB/s each when in multipath “wide” mode.
I have four of these in one of my servers in a striped set of two mirrors, and they perform very well.
Where things start to fall apart in my experience is with NVMe, where ZFS just can’t seem to keep up. Its still faster than SATA SSD’s, though I’m not sure how much faster than the SAS SSD’s it is. if I had to wager a guess I’d say “not much if at all”.
Hot spares are a waste of resources if you have easy access to the server IMHO.
Same is true for my OPNsense.
Yeah, depends on the workload but yes. Checksums and CoW on write does come with a perfomance penalty in some workloads. But than again, I see people using CEPH for DBs, so my guess is that performance isn’t the most important metric for many 
Agreed. If you have easy local access to the machine, you can usually order replacement drives fast enough that your redundancy saves you if you use RAIDz2 or above until the replacement drive gets there.
But even if you want to have spares for a sense of security, I’d argue its better to keep them cold, so they don’t wear (and use power) by running when it isn’t needed. Then you can just easily pop them in when they are needed, and you have a brand new drive to take over.
Just make sure you do a thorough test on them when you get them, so you don’t find out that they are bad after they have been sitting on the shelf for a few years when you need them.
Ceph is a different animal. If you are seriously considering Ceph in your stack you are looking at a storage cluster and require storage HA for all the VMs and whathot, that zfs simply cannot do because it’s not a network/cluster filesystem.
So yeah there can be pretty big tradeoffs depending on what the “acceptable performance” line is
Yes!
Another caviat is here:
ZFS sucks on deleting files. So if scenario contains regular purging files (blocks) I’d consider XFS.
Sorry, it’s resilvers that see a steady 800+MB/s
Scrubs are a fair bit faster:
pool: zfshome
state: ONLINE
scan: scrub in progress since Sun Oct 11 00:24:03 2026
96.3T / 104T scanned at 1.98G/s, 90.9T / 104T issued at 1.87G/s
0B repaired, 87.08% done, 02:03:16 to go
That’s from a pool with two data VDEV’s with 6 spinning hard drives each.
I guess my take when it comes to fragmentation is that if most of the data you write, you write once, and then just kind of leave it there, and you don’t go below the 80% use of space recommendation, fragmentation doesn’t wind up being a huge problem.
Would it be nice to have the ability to defragment a pool? Sure. But it doesn’t wind up being enough of a problem that it offsets all of the benefits of ZFS. At least for me.
I don’t think I’m making my point clear, when ZFS experienced an unexpected power outage it became unusable afterwards, no further access to data until forensics came in, the other file system was more resilient in this situation and was able to handle unexpected power outage and come back online.
The reason I’m mentioning this is because it would directly factor into a manufacturer’s willingness to support ZFS at the user level through whatever helpdesk system they had to offer.
That makes more sense since ZFS is supposed to implement scrub reordering in memory to help alleviate the performance impacts of fragmentation.
There’s also an analogous resilver operation but I’m not sure if it is invoked as a default, it eats up a ton of memory.
For comparison, I’m getting a 2,807MB/s scrub speed when I adjust my non-ZFS array of MG08s to 12 hdds of width.
Yeah, If curious, this is the configuration:
config:
NAME
zfshome
raidz2-0
16TB SATA Seagate Exos X18
16TB SATA Seagate Exos X18
16TB SATA Seagate Exos X18
22TB SATA Seagate Exos X22
16TB SATA Seagate Exos X18
16TB SATA Seagate Exos X18
raidz2-1
16TB SATA Seagate Exos X18
16TB SATA Seagate Exos X18
16TB SATA Seagate Exos X18
22TB SATA Seagate Exos X22
16TB SATA Seagate Exos X18
22TB SATA Seagate Exos X22
special
mirror-4
2TB Gen3 NVMe Inland Premium (Generic, MicroCenter Brand, Phison E12)
2TB Gen3 NVMe Inland Premium (Generic, MicroCenter Brand, Phison E12)
2TB Gen3 NVMe Inland Premium (Generic, MicroCenter Brand, Phison E12)
logs
mirror-3
375GB Gen3 NVMe Optane DC p4800x
375GB Gen3 NVMe Optane DC p4800x
cache
4TB NVMe WD BLACK SN850X
4TB NVMe WD BLACK SN850X
In my initial post I think I was thrown off by the resilver speed from when I was replacing a failing 16TB drive with a 22TB one last week since it was likely only showing performance data from the one VDEV being resilversd, not the whole pool (it would have no reason to read data from the other VDEV not being resilvered). That, and there is the bottleneck of the write speed of the one drive.
This pool - of courser - does benefit from not having to seek in the data VDEV’s for metadata, since the Special VDEV handles that.
To prod the original question a little: What do you guys think the reason for defaulting to btrfs is for Fedora Linux? Why not zfs for a single drive if the snapshot functionality is superior? Are the resource demands of the two so different?
(also valid for Suse/OpenSUSE)
btrfs features cover their requirements (mostly snapshots with snapper and volumes) and is in-tree for linux kernel. And also does not have strange and possibly dangerous licensing issues.
Other distros like Ubuntu have integrated ZFS because they want to “be different” but didn’t do much besides allowing it in the installer.
CachyOS is probably the most well-integrated of the bunch. But they are a secondary distro run by a community, not sponsored by a corpo like Fedora/OpenSUSE, so they have a lot smaller profile for potential legal issues. (like sure you are gonna sue the 5 lead devs, where at least one is russian and dgaf and some others might be in other countries where they also dgaf)
It’s also a timeline/historical decision, when they chose to switch to btrfs, the situation was different and it looked like btrfs could really become “linux’s own ZFS with blackjack and hookers”, which as we now know, didn’t really pan out. And I still scream at clouds daily about it.
Imho, no. As long as you don’t enable deduplication (either the “old version” or the “new and improved version”) they are in the same ballpark
I think the primary reason has nothing to do with capability or performance, and everything to do with license differences.
I’d argue that ZFS is superior to btrfs in almost every way related to capability, performance and reliability, but due to Sun/Oracle shenanigans OpenZFS is stuck on an incompatible license that causes consternation for the kernel/distributions.
Linus Torvalds has been pretty vocal about his distrust of Oracle and the license issues that cause problems, and has suggested that no one should be using ZFS on Linux as a result. While I agree with him that Oracle is a very slimy company that cannot be trusted, and the license issues are real, I’d also argue that ZFS is too good to be thrown out with the bath water as a result.
This might fill in some gaps;
So in summary, the downsides seem to be:
Those tradeoffs don’t seem to make much sense for single drives or collections of differently sized drives. The benefits are best realized when you can get substantial quantities of matched drives. That’s probably the biggest hindrance to home server use, with how many people are having to dumpster dive to get affordable hardware.
No, this isn’t generally true.
Can do it just fine.
Nahh, also false. Yes, for blockstorage and with HDDs peformance will probably dramatically suffer above 80% usage. But it will probably start to suffer even above 50%. For datasets, and for big pools, even having it 95% full isn’t an issue.
That is a plus, not a negative. No matter if you like Oracle or not (I hate that company and hope they go bancrupt with their AI neocloud deals), ZFS is also called the billion dollar file system. There were huge investments and it is battle tested.
ZFS makes perfect sense for a single drive. That is why even applicance stuff with one drive uses ZFS. Like OPNsense.
Lots of weird urban legend or niche reasons and no one talking about the real pain in the ass of using ZFS specially on Linux that is kernel compatibility. You can’t update the kernel until the openzfs folks release code for that version. Specially annoying if on rolling release.