I’m currently using Btrfs RAID6 14 bay array and I’m slowly getting annoyed by how slow this thing is. It has 2x10G NIC and 32gb RAM so when I’m copying files to it, this thing GOES, goes goes and then hits concrete wall when it runs out of ramcache. I used to run dmcache backed Btrfs RAID1 in the past but well, it was significantly simpler case since I could just get one ssd per one hdd and set it up 1:1. With this RAID… it’s not so simple and I believe partitioning 2 mirrored SSDs into 14 partitions for each HDD sounds like… Very Bad Idea™ as it’d most likely severely screw up Btrfs self-healing capabilites. Especially if one of those mirrored SSDs would die.
So at this point I’m kinda wondering what my options are. If I remember correctly ZFS supported something like RAID1 ssd cache for ZRAID array but I’m not exactly a ZFS expert so my knowledge may not be really accurate.
To give some context - it’s backup RAID array but I’m backing up VM images to it which are typically quite big but smaller than one physical SSD (1 - 2 TB, rarely more). Backup source is my workstation which has 2x25G and NVME Gen4 RAID1 storage so copying those VM images over NFS feels like eternity
ZFS doesn’t have a true write cache. There is the ZIL, or ZFS Intent Log, that can be stored on a separate fast device like a PCIe SSD, but this does not cache the incoming write data in the sense you seem to be describing.
For more performance in a heavy sequential write workload like you have here, you need to be using sets of mirrors in ZFS, and probably in Btrfs too (I’m not familiar with what options Btrfs provides). Parity-based RAID is terrible for heavy write performance in general.
RAID5 and 6 are marked “unstable” in the BTRFS docs, so it’s probably not a good idea to keep your data on that pool for long. I understand the new 6 series kernel has major improvements for BTRFS+RAID5/6, but still not declared stable enough for production use. See also:
It’s okay, I keep my data on this Btrfs RAID6 for like 8 years now or more. It’s still on old 2TB drives, so sounds pretty safe to me. It’s just that now I upgraded to NVME in my workstation and this old HDD RAID all of sudden started feeling really slow in comparison.
14 drives, even old ones in RAID6, should be able to saturate a 10Gb network, at least when sequentially transferring large files. What kinds of speeds are you actually getting? How does the CPU load look, etc.? There must be a bottleneck somewhere. OR you are seeing caching on the SENDER side rather than on the NAS!
7GB/s gen4 SSDs (which will throttle down to 1-2GB/s any way after a while!), will just saturate 10Gb (=1,25GB/s) just the same.
I’m getting around 850 MB/s read and ~450 MB/s write sometimes above 550 MB/s. It strongly depends because I’m using zlib compression and LUKS encryption so performance varies but on average it’s around that. With smaller files it’s much worse.
I’m using 2x FireCuda 530 4TB in workstation so they’re quite good at sustained write performance (I was specifically looking for that in NVME drives). CPU load on HDD RAID machine is around 60% during backups so it’s not low but it’s not all that high either I think.
Raid is broken in BTRFS (you can read it in plain English in BTRFS documentation). It ate my data many years ago, and it will eat yours. It is not a matter of “if”, but “when”.
ZFS doesn’t have “write cache”. It has an optional SLOG for synchronous writes.
