I tend to prefer ZFS. Its protection against silent corruption is second to none, and its snapshots and command line management interface are amazing. Backups are also really convenient using the zfs send/recv function and sending stuff to another ZFS box. But it is not the right application for everything.
ZFS likes lots of RAM to perform well, and it is generally a good idea to use ECC for it. It also does use some CPU (though this is less of a big deal today with our modern CPUâs than it was years ago)
You would also not want to use ZFS for swap space, as it can create a race condition if you do. (RAM gets low, â greater need for swap space â ZFS ARC in RAM grows to accommodate more use â RAM is even lower â greater need for swap space â etc.)
BTRFS is much like ZFS, but I honestly donât have much personal experience with it. I keep reading up on it to see where it stands, and keep hearing of people having problems.
MDADM is an OK traditional software RAID solution. CLI management is so-so, and can be a little counter-intuitive. I have a two way MDADM mirror on my all in one NAS/VM server that only does swap (because of the ZFS issue mentioned above) but honestly it sits empty most of the time.
ZFS can be hit or miss in the performance department. It tends to perform really well on hard drives and SATA SSDâs. Essentially any drive that is slow enough to allow for CPU operations while waiting for the drive will perform amazing on ZFS. This is especially true for traditional hard drives, where its copy on write block based design helps overcome the seek limited random reads of drives, by essentially converting everything into sequential reads. It was created in an era where traditional spinning hard drives were the norm, so the code kind of expects it to have ample time to do CPU computation while it waits for the drive.
This means that when NVMe drives are used as data drives, ZFS does not perform as well as expected. While it will still be faster than on hard drives or SATA SSDâs, oftentimes, even on slower Gen3 NVMe drives, if these drives are used in the main data pool, you can get performance below what youâd expect from a standalone drive.
ZFS can - however - utilize NVMe drives as caches and so called âspecialâ vdevs (for metadata and small file storage) very well. These appear to be implemented in a way that is aware of modern high speed storage, and thus donât appear to be held back as much by the problem where the CPU expects to be able to take its time while waiting for the drive.
Writes can also be sped up a lot by using special purpose low latency devices as SLOG/LOG/ZIL devices.
But this tends to mean that ZFS is best when used in larger more complex pools, that have large amounts of RAM, mass storage over SATA or SAS, some dedicated special devices that are NVMe for metadata and small files, some low latency devices (like Optane or battery backed RAM devices) for SLOG/LOG/ZIL and maybe even some read cache devices (which could also be NVMe) to speed up reads of frequently accessed files, while offloading the main data pool to do other stuff.
That said, you donât have to have a large and complex ZFS pool to make use of ZFS. I have plenty of systems that boot off of simple two-way mirrors. Just donât expect it to be a high performance solution with two NVMe drives. Many people have the expectation that a basic two-way mirror is going to have twice the read speeds of a standalone drive, and with NVMe in ZFS this will simply not be the case. Write speeds will also be comparatively slow (at least if they contain sync writes, which they likely should if you are using any kind of database)
So why do I still use ZFS this way? Easy. In those applications, performance is not the priority, and I love the reliability, flexibility and management of my ZFS pools.
What would I use instead if performance were a priority? Not sure. MDADM is not a performance screamer either. I simply donât know enough about how BTRFS performs in this situation but from what I read there are enough issues that I am not ready to use it. At least not yet.
ZFS decked out to the nines with special devices, gobs of RAM, SLOG devices and cache devices can have great performance for things like databases and the like, but in a basic two-way mirror form, you will not be impressed by the performance. It wonât be terrible. It is totally usable, but it also wonât impress.
I think in applications where high performance is crucial out of a simple two drive pool, Iâd probably just use standalone drives to be honest, and just have something set up to do regular backups.
The thing is, RAID is not backup. It helps you keep a greater uptime if something goes wrong, but it is never intended to replace the need for backups. You should be backing up either way.
Which brings me back to the advantage of ZFS send/recv.
If your other NAS you mention runs on ZFS it would be stupidly easy to set up backups from any ZFS based pool anywhere on the network to just send incremental block-for-block backups to the main NAS.
Anyway, I hope this helps.
