BTRFS setup with 4 drives (WIP)

Hardware Hub Build Logs
, , ,
1

Well, it’s been a long time coming, but I finally maxed out on my drive capacity. In a previous post a few years ago, I posted about getting a second 4TB drive for my home server. Today, I want to present my soon-to-be setup.

What I got

My current home server runs off a Dell 5070 micro (a 1L PC) with 500GB of NVMe for the OS and VMs. The bulk storage is in a DAS box from Orico Honeycomb 4-bay drive box, with 2x 4TB WD Red and 2x 12TB WD Red Plus.

How I put it together

Since the journey began, I played with some options, and found BTRFS to be the best choice. It has small overhead for the computation, does not require ECC memory AFAIK, and is overall a good little filesystem. I mirrored the two 4TB drives and they work just fine.

As for the data, I mostly use the drives for family photos, movies and shows, everything pointed to multiple docker containers. These two drives contain everything and are close to full.

With the added 12TB drives, I have a new plan in mind.

Plan 1: Move to new, keep old for snapshots

As the new drives are, well, new, I can move all of the stored stuff I have to them, and just remount the big drives in the old drives’ place. This way I don’t have to mess with my VMs and set everything up again.

I get triple the storage easily, letting me use the old drives for stuff like snapshots or media I don’t really care about (my movie collection comes to mind). Alternatively, I can keep media on the new (as it’s big) and have a second copy of the critical stuff on the 4TB drives (photos, documents, etc.).

Plan 2: Create a 16TB mirror

The other, I guess more complex plan would be to create something of a striped mirror: A mirror of one 4TB and 12TB drive, giving me 16TB of mirrored storage. This is a thing I could do , but I don’t feel I should. Let me explain.

The 4TB drives already have some mileage on them. If I attach them in a pool with the new drives, my risk of losing stuff is bigger, as they will contain bits and pieces of critical data. Sure, they haven’t failed yet, but it’s a matter of time.

Question to readers

What version would you go for? Would you separate based on data stored (photos vs. movies) or would you go the other way and create one consistent pool? I don’t really know how to add drives to the same BTRFS pool without losing data, but I’m very open to suggestions! So far, using the 4TB drives for snapshots seems like a good idea, maybe removing them from the mirror and getting 8TB of “disposable” data storage would work, too. Something like 2 movie/series drives and 1 12TB mirror for the important stuff (VM backups, photos, regular backups, you know what I mean).

This has been a short summary of what I’m up against now :slight_smile: Thank you for reading!

2

Since you didn’t mention any separate backup location in your post I assume there is none.

While mirroring drives mitigates the event of a (single) drive failure, it is not a backup, which mitigates a long list of other negative events (fat-fingered user, malware, etc.).

For that reason I would strongly advise to exercise “Plan 1” as it creates a physical copy. It’s still not a backup in separate hardware not in a separate location (google “3-2-1 backup rule”) but it’s an improvement from your current setup. At this point you’re a hapless rm -rf away from losing all your family pictures.

btw. There is no requirement for ECC ram. It just doesn’t exist. ECC mitigates against random errors occuring in memory. This situation happens to all computer systems given enough time, but is extremely unlikely in typical desktop use, which is why “the industry” doesn’t deploy ECC RAM in desktop systems (prioritizing cost savings over risk mitigation).

1 Like
3

Not per se if he is using BTRFS snapshots; those are read-only and can only be deleted by running btrfs snapshot delete ...

I would probably do BTRFS RAID 10 on all four drives, and later down the road replace the 4TB drives as needed. That would protect you against one drive failing.

In that case you btrfs add ... and then btrfs balance start -dconvert=raid10 ...
https://btrfs.readthedocs.io/en/latest/Volume-management.html#add-new-device

This kind of flexibility is IMO one of the strengths of BTRFS

I’m not sure how exactly you plan on doing this but the low overhead snapshots of BTRFS can only be done within the filesystem; otherwise you’ll need to send/receive the snapshots or use a tool like btrbk. So I’d opt to having a bigger RAID 10 pool and keeping backups on another system/external drive.

4

Damn, I forgot to mention that :slight_smile: I have an off-site backup with my parents’ house (that’s where photos are going every night, currently). This saved my behind quite recently, because after putting the two new drives in, my BTRFS mirror of the 4TB drives just… disappeared :smiley:

But yes, I do have an off-site backup from my house to my parents (and vice versa). This is off-site. Sure, I’ll work on making a second backup location, but as it stands now, I don’t have a place to put them (except for using something like Cryptomator and uploading the photos to some cloud storage, but that’s not my point at the moment).

So far, I think that keeping snapshots on the older drives (and monitoring them) may be the way to go. It would allow me to “undelete” the files if I mess up.

1 Like
5

That’s what I was afraid of, I wanted a way that would allow me to mirror snapshots to a different drive, but I guess it’s not going to be a backup on the 4TB drives, an off-site backup to the parents’ house, and I’ll hope for the best. The 4TB pool itself is mirrored, so one drive failure is okay, the 12TB pool is the same, so it should work well enough.

As for the RAID 10 on all 4 drives, I’m afraid I don’t have the storage space to move my files to in the interim. Is there a way to enlarge the old 4+4TB with the new 12TB drives?

Thanks for the comment!

6

I don’t know what you were doing to cause your mirror to disappear, but be aware that dd’ing entire partitions is not a valid way to copy mounted file systems. When the kernel catches wind of the fact that you have another drive with the same uuid, then btrfs will start using either drive semi–randomly and this will definitely torch your file system.
Also learn to use snapshots and btrfs. They are an absolute Joy, a low overhead way to keep an historical view (not a backup).

2 Likes
7

It makes little sense to mix and match different sizes, while you can concate two drives you’ll get rather poor performance for little gain. Just keep both as plain mirrors.

I know this is a “holy” topic but what’s your reasoning between btrfs and zfs?

8

How does your /etc/fstab look? Probably device paths simply changed so the OS can’t find the proper drives any more. Best practice is to use UUID’s for mounting cf.

https://wiki.archlinux.org/title/Fstab

You can see how the UUIDs are mapped to the ‘usual’ devices by running ls -l /dev/disk/by-uuid. Also use lsblk to see which partitions sit on which disk; probably a hint as to which drive is which?

You definitely can, but it involves btrfs send and btrfs receive, and you need to provide the proper ‘parent’ subvolumes if you want to make use of the efficient snapshots… It is complicated to do manually. Btrbk is a tool that is nice to do this with (I use it for my backups to my NAS). But if the drives are literally in the same system I don’t really see the value of this over a RAID 10 setup.

That’s not needed. BTRFS can do this online, on a running system. You can even reboot and resume the process. I added a third drive to my pool a few months ago the same way. The system keeps running and is fully useable (albeit with a bit lower disk performance due to the balancing in the background). The way to do it is what I linked in the previous comment…

https://btrfs.readthedocs.io/en/latest/Volume-management.html#add-new-device

YES! Also beware of dd’ing BTRFS in general (even offline). BTRFS uses partition UUIDs to recognise devices in the filesystem, so if you dd and have the same UUID multiple times it will refuse to mount in the best case or perhaps worse things can happen.

I don’t see an issue with this. I wouldn’t run a production database like this but for bulk file storage it’s absolutely fine.

9

…until something breaks in some weird way and you’re likely going to see reduced performance as well.

1 Like