I am using the opportunity to setup my main work system from scratch and I am wondering about which file system and raid setup would be perfect. I previously ran two Samsung 980 Pros in RAID 1 with BTRFS. Since I got a little low on space, I got two more.
As BTRFS really steers you towards RAID 1,0 or 10 and RAID 5 is rather unstable from what I read, what are the best options for me?
As a distro I am mostly siding with Pop, but might go with Ubuntu instead if it provides a benefit.
My main goal is stability and reliability for my main work system, on my personal system I just yolo it with a single NVMe with BTRFS.
The options I am considering so far are:
ZFS Raidz1 (giving me 6TB usable out of my 4 2TB NVMe drives)
MDADM Raid 5 with EXT4 (giving me 6TB usable out of my 4 2TB NVMe drives)
MDAMD RAID 10 with EXT4 (giving me 6TB usable out of my 4 2TB NVMe drives)
BTRFS Raid 10 (giving me 4TB usable out of my 4 2TB NVMe drives)
Is there something I am not thinking about? Anyone who runs a similar setup and why did you choose it?
Bonus question: Any opinions on whether pure Ubuntu or Pop!_OS might be more reliable? (With Nvidia drivers Pop seems more favourable to me, but again, maybe there’s something I am missing)
I would discourage you from using ZFS. BTRFS is integrated into the Kernel and ZFS, because of the unique licensing, probably never will be in Linux. I lived with ZFS as a filesystem for a while but it makes trouble on a regular basis. In case you rely on precompiled binaries you always need to make sure the installed ZFS packages are compatible to the kernel and in case you use a kernel module that build automatically when you install a new kernel, you will need to fix it yourself when it does not compile and errors out one day. Definitely not a recommendation.
BTRFS does fine if you ask me, but I only ever did mirrors using it.
MDADM is high performance and rock stable, but it does not offer features like checksumming and compression on it’s own. I figure MDADM with BTRFS on top should be doable, but I have no experience with it!
I agree with @H-i-v-e about shying away from zfs in your case, but for different reasons.
zfs shines in providing high-capacity, high-performance, HDD-based storage. This is simply not your use case (not HDD, 4-6TB usable storage is arguably not high-capacity in 2024).
Otherwise, zfs is very stable and even dkms-based upgrades have been working flawlessly for quite some time.
You need to make up your own mind on the licensing debate.
btrfs offers a similar feature set compared to zfs and is a modern, feature-rich file system (checksumming, COW, snapshots, send/receive backups). I would take a closer look, especially if you plan on using these new features.
My personal experience with btrfs is mixed with experience of recent fs corruptions. Still running it on multiple systems, though. I would absolutely shy away from raid5/6 configurations based on btrfs (you do).
+1
Maybe the right choice if you don’t plan on being or becoming a file system geek
How about upgrading to larger capacity NVMe SSDs? To avoid all this hassle altogether?
Currently, I am running BTRFS on all my systems and was happy as well. Used BTRFS Raid 1 on my main system until now, but my home subvolume kept filling up and I wanted to switch away from Elementary as my main distro for work.
The main reason was that I wanted to get this done tonight and I could get two extra 2TB 980 Pros for four in total today, while I would have had to wait for 4TB 990 Pros much longer and then I already need it to be up and running.
I think I’ll do a Raid 10 BTRFS setup for now, using Timeshift and timeshift-autosnap-apt for extra rollback abilities.
Ultimately, I still want a Raid 1 setup at all to minimise interruptions due to (admittedly rare, but not impossible) drive failures
*If your distro is . No offense here, but I’ve been running ZFS on Proxmox, Void, NixOS and FreeBSD and none gave me any headaches (although the first and last treat ZFS as first-class citizens).
Never had an issue with zfs-dkms.
That said, I can’t vouch for ubuntu or pop, so YMMV.
Have you considered building a small NAS? Given you’re using RAID1, it looks like you’re looking for reliability more than speed. You’d probably be better off building a small NAS with something like an odroid h3, a type-3 case and slapping 2x 8TB QLC sata SSDs in it, in mirrors. If you don’t need speed, bulk 16TB+ spinning rust in a mirror (and a type-1 case instead).
You could also go for big number of small drives and grab an asustor flashstor 6, but can’t vouch for this as I can for the h3. NVME form-factor should have more life than 2.5" sata will, meaning next-gen m.2s should be about as large as sata ones.
Is there a particular reason you want to add more drives to your existing rig, instead of keeping everything outside of it? Do you have a particular need for relatively fast local storage, or need to have the data with you if you travel (although if it’s a desktop that’d be a bit questionable)?
yes, my home directory alone was 1.4TB now; I like the additional speed, but I wanted at least a RAID 1 for redundancy.
In fact, I have multiple NASes at home… and a CEPH Cluster… and another 8TB NVMe in this very system as fast scratch disk for temporarily parking ML models, datas etc. It just got a bit tight on my home partition, that’s all. My main two concerns is that no matter what happens I can keep working if I am in my flow (including disk failures) and to increase space, hence going from RAID 1 to RAID 10
I just finished my install using BTRFS RAID10 and the additional snappiness is nice. If the upgrade from Threadripper 5000 to 7000 wouldn’t cost the same as another high end GPU, I’d almost be tempted just to switch to gen5 storage… but I’d rather have an RTX 6000 ada…
TBH, if you’re already using BTRFS, there’s not much reason to backup, format everything and restore, instead of just expanding to RAID-10. Probably ext4 (or xfs if you work with really large files) might give you better performance with md, than btrfs.
And it might not be negligible, but it sure would be a PITA to switch now. But LVM snapshots suck (not that you couldn’t just keep root on btrfs if you had 2 partitions on each drive, like /dev/nvmen2 for btrfs and /dev/nvmen3 for md, but that’s just too horrible for my taste, I’d rather deal with single slower FS than split FS on multiple drives).
You’re not going to get a noticeable performance in every day workloads, with either. Save yourself the hassle, use what you’ve currently got. BTRFS is a great filesystem, as long as you don’t choose raid5.
Pure Ubuntu vs Pop basically boils down to 2 questions: Do you run Nvidia? Do you use ZFS? Here’s your OOTB compatibility matrix:
Ubuntu
Pop
Nvidia
ZFS
Given my previous advice, I’d recommend you just stick to Pop and enjoy it. They’re both basically the same OS.
+1 for BTRFS. I have only been running it for about a year-ish but it’s been great on my laptop and daily driver desktop (non-gaming). I used this guide to set it up.
Edit: Forgot you have multiple drives, I used the above guide along side this guide to get RAID1 running (which can be converted to whatever you want, RAID10/RAID0 etc.)
I highly recommend btrfs for the ease of use from my personal experience.
I use btrfs on both my main drive and additinal data drives on 2 machines.
I have btrbk set up to automatically snapshot my subvolumes I care about and send them to my external drive as well as to my other machine. This ensures my snapshots are in several places on 2 machines. Fairly high redundancy. I am happy with this level of safety and ease of installation.
I turn off COW on certain subvolumes, eg games and certain data subvolumes.
Since subvolumes are very cheap abstractions, I believe they should be used liberally to customize how my data is dealt with. A home subvol need not be monolithic. For example, I /have @home mounted on /home and @games mounted on /home//Games
For ZFS, there are solid guides for setting up root on ZFS on Gentoo, and as already pointed out, ZFS is supported on, at least, ubuntu.
Btrfs is pretty simple if you don’t use all its features and you can basically discover those features later on so you’re not punched in the face with requirement to know about all fs concepts. I’m using btrfs since very early days back in 3.x kernel era (when btrfs was still highly experimental) and it’s really fine. I’ve been using it all configurations - single, dup on single drive, raid1, raid5, raid6, raid6 with raid1c3 metadata… zlib, lzo, zstd compression, all this stuff. It’s fine.
Personally for NVME Gen 4 storage I’m using RAID1 with LZO compression and snapper managed snapshots and it’s really fine. Of course RAID1 on btrfs level.
Also side note regarding TR 7000 - imho motherboards for 7000 are exceptionally meh. I’m on TR 5000 myself on ASRock WRX80 Creator and current TR 7000 motherboard offerings are far from perfect for my use case (quite literally, starting from form factor lol).
. No offense here, but I’ve been running ZFS on Proxmox, Void, NixOS and FreeBSD and none gave me any headaches (although the first and last treat ZFS as first-class citizens).
