BTRFS Raid Confusion

Hardware Hub Storage
,
1

Since I moved my office back home I’m getting buried in USB drives. And rather than buying more and more usb portable things, I’d rather just set up a server and be done with it. Then I’ll setup offsite backups next

So I’m building a backup server. It won’t even be on most of the time. I’ll turn it on, run a backup of the systems I have, and then turn it off. Run a scrub periodically and be done with it. The OS for the server will be on a separate drive than the raid array (no root-on-raid).

My initial thought, was slap a copy of debian on it, mdadm raid 5 or 6 and done. But then I was able to snag me some 20TB exos drives; and I’d expect the recovery time (if needed) to take forever…

Truenas and ZFS just seems like overkill, especially since I won’t actually be working off the nas. But from what I’ve seen, if using zfs, truenas is the best way to do it (rather than install zfs on debian).

Obviously mdadm raid is always an option.

On the btrfs side, (and yes I could install btrfs over mdadm, but why? I don’t really see an advantage over ext4) it appears that native raid56 should still be avoided? I’m still kinda fuzzy in understanding what they mean in their docs though. From what I can see the “unstable” stuff mostly applies to “zoned” raid - whatever that is.

https://btrfs.readthedocs.io/en/latest/ch-volume-management-intro.html

https://btrfs.readthedocs.io/en/latest/btrfs-man5.html#raid56-status-and-recommended-practices

You can see the confusion in that I’m having an issue even formulating a coherent/discrete question.

It appears that btrfs is currently best suited for a raid 0, 1, or 10 and zfs for parity style raid 5/6. Am I reading things correctly?

Depending on what happens in the future I don’t expect shrinking the array. If anything expanding. And it looks like both btrfs and zfs will do that when they are running the raid, rather than mdadm.

Any other advice/wisdom? Thanks.

2

btrfs is great but “The RAID56 … should not be used in production, only for evaluation or testing.”

Truenas is probably perfect for you but also consider

1 Like
3

I’ll check them out. Thanks

4

I’ve had no issues running Raid5 with BTRFS on my Unraid server for my backup pool. My drives are all the same size, I think that helps avoid the write hole since it’s not hard to compute where the parity block needs to land but IDK.

5

Ive used raid 1 with BTRFS for nearly a decade… No complaints…at the moment .
Total devices 5 FS bytes used 15.74TiB
devid 1 size 7.28TiB used 5.08TiB path /dev/sdf
devid 4 size 5.46TiB used 3.43TiB path /dev/sdc
devid 6 size 10.91TiB used 9.23TiB path /dev/sdb
devid 7 size 9.10TiB used 7.51TiB path /dev/sda
devid 8 size 10.91TiB used 9.38TiB path /dev/sde

Even with random drives.

1 Like
7

I know this is going to sound very vanilla on this site but I just rebuilt my media server like a week ago and went with Proxmox with qty (6) used 6TB HGST drives in RADIZ2 for 24TB of usable space…I then created an SMB share directly from Proxmox and share it to my VMs. Some folks would pass individual disks or pass an HBA card through to a VM, that would have worked, too, but this was easy for me and gives me some freedom, and the performance losses of SMB are inconsequential for my use case.
As for expansion, I MAY add in another RAIDZ1 array for qty(3) 4TB disks and if I do then I will use MergerFS to combine those two ZFS pools into one virtual pool in Proxmox and then share that MergerFS through SMB. The nice thing there is my clients won’t care - theyll just see that the pool is bigger (or smaller) one day. MergerFS is great.
Anyway probably not what you’re looking for but I thought I’d share what worked for me.

1 Like
8

I only quoted the devs, I would love it if the project got to the point where they feel their RAID56 implementation is suitable for production.

9

I use btrfs raid 1 on one of my unRaid nvme cache pools for almost two years, zero issues.
It is generally considered best as you said not to choose btrfs for raid 5 but rules are meant to be broken sometimes. There can be use cases where a non-standard design is the preferred option.
Love tech, 10 ways to skin the cat.

1 Like
10

Sokay. I appreciate it. I keep forgetting about proxmox.

11

So it looks like the ultimate decider here is basically going to be what kind of raid level I’m going to set up.

BTRFS seems to have a “lock” on 0/1/10 where btrfs raid 1 (and thus 10) is interesting enough to play with given how it does raid 1 - where apparently things are mirrored on different drives (??) rather than across all of them?

ZFS / Truenas on the raid 5/6 I’ll need some more research on how it handles expansion. Make the zpools small enough and it gets “easy”. Minimizing the “tax”. But will that let me do what I want later? Not sure. More research is required.

Something tells me that, regardless of what I decide next weekend when I build this, I’ll be revisiting it in the future.

Thanks folks

12

Wrote a guide here a while back…

… needs some updates, but all the broad strokes stuff should still be fine.

Practice setting it up in a VM, because why not, … and go from there.

I’m auto updating in the background both my NAS and my router, that are both based on that setup, they’re on the testing track, and I’m not going to lie, about once a year it won’t reboot and I need to come in with a flash stick and fix something.

Haven’t lost data with btrfs since … about 2015 iirc.

I also have another jumble of very large drives, running mergerfs, snapraid and samba. It’s a very very surprisingly good working setup for lots of media and archival storage and secondary backups. Adding a drive and rebalancing using rsync of all things is a bizarre experience, … but it works.

How many drives do you have?

1 Like
13

Thanks. I’ll check out the guide - didn’t even think to look there tbh.

At the moment 4 drives. I might just say f it and get some more – got some accounting to do first. Which (of course) will also dictate what I do.

14

Unraid is it realy btrfs raid ?

15

I tryed unraid and it would load a btrfs raid of drives

16

In Unraid when you create a “pool” (what used to be the “cache” drive is now just a generic pool since you can create more than one) it will use BTRFS by default, however, you can change it to ZFS if you wish. The normal array drives I think use XFS by default.

1 Like
17

BTRFS RAID 1 simply stores each block twice on different drives. On which drive exactly is kind of random. RAID 10 does the same, while striping across the other drives.

So when using RAID 1 on btrfs you don’t have your disks ‚mirrored‘. No two disks have to hold the exact same data. But since each block is guaranteed to be duplicated across two disks you can lose one disk and still be safe.

18

Thanks for that, because I was still trying to wrap my head around btrfs raid 1. Good explaination.

As for this project, my grandiose plans have had to change somewhat. Buying the hardware to set up a server and maybe splurging on a few more drives (despite the credit-card interest…) will have to be put on hold.

On the plus side, my shiny new water heater / water softener / water filter all look really nice and happy in my garage.

Ah, the many joys of home ownership…

So the new plan is…

Take my 4 drives, install them in my desktop (4 sata ports avail) and set them up as btrfs raid 10.

Then, since I already have one of these:

I’ll move them over to there. and can do my future backups that way. Leaving everything on the mobius set up as jbod shouldn’t break anything raid-wise. Hell, in my experience it doesn’t even change the mount points for the drives.

This allows me to swap the drives out keeping the “relative safety” of a disconnected system in case of a random powersupply fryup.

19

Sorry for necrobump but I’d just like to mention that I’m using BTRFS RAID6 since the very first release (kernel 4.6 around 2016 I believe? maybe earlier) and I did loose array only once and it was with old, fatally flawed implementation (I did not loose any data though, it just started flipping to read-only mode and I had to build new one and migrate data). It started as 3 bay RAID5 and then I kept rebalancing it over and over and over again to 14 bay now (14x2TB). Ability of btrfs to freely change RAID profiles and extend RAID is amazing, afaik ZFS only recently added this functionality.

Performance is okay, I’m getting around 800 MB/s read over 10G connection to NAS but I’m using LUKS and reading 14 encrypted drives at once is quite taxing, CPU gets hammered pretty hard.

RAID56 in btrfs is fine. Devs are really careful with wording because there was fatal flaw that could trash your array while doing scrub in old implementation (I believe my lost array was affected by this issue, what’s worse it was unfixable for existing arrays, you had to make new filesystem because there was no in-place disk format data conversion implemented) and not many people use it so they play it very safe. Btrfs really struggles with reputation of unreliable over complicated thing so they really try to focus on stability and raid56 doesn’t help them with that.

But I’m using btrfs exclusively everywhere since kernel 3.x back in like 2013 or something and I did not loose any data (though I did loose few filesystem on laptops because back in the early days it really didn’t like kernel panics and dirty shutdowns, now it’s much better tho). I also use btrfs at work on servers that we use in production in big companies as integrators. And it’s fine.

1 Like
20

imo BTRFS native Raid 6 is a complete nogo and I would strongly discourage raid 5 use. However BTRFS on top of mdadm 5/6 is serviceable.
These are the main issues currently affecting the reliability of BTRFS raid 5/6 in my eyes:

  • BTRFS raid5/6 scrub performance makes it unsafe, taking weeks when it should take <24 hours because of how it is implemented

  • Power failure or kernel panics will still corrupt metadata if on raid 5/6; supposedly a metadata journal is going to eventually be implemented to help with this issue but it hasn’t been developed yet as I understand

  • BTRFS ENOSPC issues still persist

  • RAID write hole still exists (to be fair mdadm also suffers from this)

  • incorrect dev stats output misleading users to what drives are failing-- I’m not 100% sure this is still an issue with the most recent build

21
  • BTRFS raid5/6 scrub performance makes it unsafe, taking weeks when it should take <24 hours because of how it is implemented

^ this is getting a bit better. I recently performed scrub on 28TB array and it took something along 4 days. It used to be over week on older versions. That said please note that btrfs raid 5/6 corrects data on read/write so I don’t really perform scrub that often.

The worst part is powerloss corruption and that’s real [F] and I gotta agree with that. Though it depends how often you experience such since it’s not like every panic or unexpected power off will damage array