Raspberry Pi 3 NAS with RAID

Hardware Build a PC
, ,
1

Given that I have a spare Raspberry Pi 3 lying around I thought about running my Seafile server on it. But then I didn’t trust the SD card of said RPi3 for my data and I thought about attaching a USB storage to it. This then led to me worrying about the data integrity and I began reading up on turning my Rpi3 ino a NAS with some form of RAID. (I know that I don’t have to have it running as a NAS to use RAID, I’ll come to this point later.)

Anway, one easy solution would by to purchase 4x1TB USB thumb drives and attaching those drives to the RPi’s USB ports. I would then (probably) install Open Media Vault (OMV) and either run BTRFS with RAID 10 or ZFS with RAID Z1.

Personally, I would prefer BTRFS but it still has issues with RAID 5/6 and I wouldn’t want to use two drives for redundancy. Hence, I was leaning a bit to ZFS but then I stumbled over the often stated RAM requirements of 1GB/1TB storage. However, I also discovered that this statement seems to be false since it was given to corporations which had a very high throughput and also high performance requirements. Given that it will be running on a RPi3 more or less throws any performance concerns out of the windows :laughing:.

Furthermore, it was also explicitly mentioned that ZFS’ RAM requirements are rather small as long as the deduplication check is turned off.

Also, since I will be having 2 or 3TB of storage available I would also store additional files and not only those on Seafile. Hence my requirement for a NAS.

I do know that I would probably have a higher benefit from building one for this very purpose or from buying a Synology thing, but right now I do not want to do that, because I will be moving in about 1y+ and my requirements for said NAS will be different then. (Currently, it prefer a very small device which is connected via WiFi, in about a year a network cable is preferred and I won’t care about the size.)

Do you have any recommendations or any kind of advice? Would you go with BTRFS or ZFS? Any opinion on OMV? And finally, how easy is it to move a BTRFS/ZFS raid? For example, if my RPi3 dies can I easily attach it to another Linux machine and restore the data?

2

Check out Jeff Geerling’s youtube channel. He’s done some NAS projects with the rpi recently which should be helpful.

1 Like
3

The Pi3 ain’t gonna cut it for something like that. I/O through USB 2.0 is painful but, if you really really want to do it try with just two drives in RAID1 and see what happens.
Any other kind of RAID configuration that requires parity calculation done on the Pi CPU is too much. Even the Pi4 can’t decently keep up with RAID5/6 arrays and only building or rebuilding such an array takes so long that you might kill a drive or the entire array doing so.

I wouldn’t touch Pi’s WiFi with a 10ft pole. I’ve never had a good experience with it, especially for file transfers. Are you sure you can’t cable it into your network?

OMV is pretty great when it comes to a ready out of the box experience that will avoid tinkering with SSH for the most part.

No, once the Pi is gone, all the data in your array are gone unless you’re willing to take the whole thing to specialists that might find a way to recover it. That’s also why I was suggesting a RAID1 array, because it’s very simple and if the Pi or one drive dies all the data are still there easy to access.

3 Likes
4

I will have a look at it, thanks! You are talking about this vid, aren’t you:

Yeah, currently cable isn’t an option.

:thinking: - thanks for this input, so RAID5/6 should be avoided makes sense. As a result, I could be using BTRFS again. Should RAID 10 work as well? From a performance perspective it shouldn’t really matter but I don’t know if the same holds for accessing the data once the device itself dies.

Edit: Here it seems like moving the array once the device dies shouldn’t be a problem:
https://btrfs.wiki.kernel.org/index.php/UseCases#Are_there_a_similar_commands_in_BTRFS_like_ZFS_export.2Fimport.3F

It’s also mentioned that ZFS has an import command which could be used if the device dies, but I assume you would have to execute export first which isn’t possible when the device has died.

5

Very unfortunate. The WiFi on those boards is pretty weak, but it’s worth trying I guess.

Never messed with it, but if it requires computation like RAID5/6 is obviously a no for the aforementioned reasons.

Since it’s a pretty simple configuration it should work.

If the array is healthy you can move all the drives to another system and just import the array, no export needed beforehand. So, if you want to use ZFS, is much easier to save the data from another machine if the Pi dies. But for RAID5/6 still stands that might need some third party to try to recover the data in case of failure and depending on the extend of the damage.

2 Likes
6

If BTRFS on your side here a typo? In this example, I’m talking about ZFS and you also mention that no export is needed beforehand.

1 Like
7

Yes, my bad. I meant to say ZFS. I corrected the post.

1 Like
8

I have 2 pi 4’s, each had a ZFS mirror, on USB attached rust drives.

Performance is fine for a single casual user, and runs Emby okay, at 1080 single stream.

Obvs. the 4 has more ram, but I’m pretty sure you can tune it down.

I have exported the pool from the pi’s, imported it to my main machine, uploaded some stuff, exported it, and imported back to the pi’s.

I ended up splitting one mirror for space, and just yolo-ing the drive.

USB devices can be finikey, and some drop out/ go to low power mode when idle, and so might cause problems for raid arrarays.

Also, the USB connection probably removes a bunch of trhe smarts they bake into the ZFS tools, with detection and stuff, but I’ve not have an error reported Yet, from 4TB of media files in the last 5-6 months.
(small sample size)

I did have one USB dock, which would spin down, dropping the drive from the array, degrading the array. It required a zpool clear and a scrub to bring it all back, but it;s annoying, and if the other drive had died, I could still swap the disks, and import -F the drive which had been dropped. as a single (degraded) drive until I could replace the actual dead drive.

All that to say, I’m very Sure ZFS will work, and if it’s just a mirror, I’m pretty sure MDADM / BTRFS would be fine.
I would again mention (for the third time) to have good USB connectors/docks/caddy/whatever

2 Likes
9

So, this means for a Z1 ZFS RAID (which is more or less RAID 5) just attaching the drives to another PC and executing an import wouldn’t work, whereas it would with ZFS’ RAID 1.

Likewise, for BTRFS no import is needed when RAID 1 is used and I can just attach the drive to another PC.

That’s good to know, I’ll keep an eye on it :slight_smile:

10

Z1 ZFS RAID will work importing it on a new machine because there’s no controller between the machine and the drives that’s scrambling the data between all the drives you’re using in a specific way determined by the controller.
At this point, since you’ll be using software RAID5/6, I’m second guessing if you’d be able to import that too on a new machine or not. Either way, I won’t try and use it.

Yes because it’s just mirroring with BTRFS filesystem. The OS is just mirroring the written data to both drives so you could even use exFAT, NTFS (if you were a masochist), ext4, F2FS etc.

2 Likes
11

Thanks again, now everything’s clear :slight_smile:

:laughing:

1 Like
12

Wait, are you talking about using BTRFS to make both the array, and filesystem?

Or an MDADM array with a filesystem on top?

Doesn’t MDADM require a —scan and —assemble or something to search for the providers, and make an array?
Then the filesystem would need to be manually mounted? Or added to fstab for permenance?

1 Like
13

MDADM + BTRFS

Create MDADM array > mount the filesystems on the drives > mount the MDADM array > automount in FSTAB

MDADM is creating a “new” device out of an array of drives so it’s pretty easy to work with it. Instead of being called /sd* it can be addressed as /md*

1 Like
14

You don’t have to format /md* after building? I was pretty sure the filesystem goes On the array?

But, it has been Years since I touched mdadm… and even then, I only used it briefly on one array.

I don’t know how it works, but I do suspect it would be better in the case of a Pi nas.

It doesn’t have ram caching and stuff, which makes ZFS great on beefy machines, but not needed on smol devices

1 Like
15

Yes, I’m THAT stupid. I was debating wether the filesystem went on the drives before or after the array. I’ve used MDADM a couple times and always had to go back to my guide and check the order of those steps. Thanks for correcting me.

1 Like
16

Pi 3… oof. I got a Pi 3 and a Pi 2. I love both of them, but wouldn’t use them for storage or I/O stuff (planning to have them to be K3s master nodes, with worker nodes being Pi 4s and Rock64/Pro64s).

If you really want a DIY ARM NAS, get a RockPro64 and add a 4 port SATA card or HBA to it. But by this point, unless you really want to avoid x86, you are better off grabbing an el-cheapo motherboard, a low-powered dual-core CPU, 4 GBs of RAM and a small case that can fit 4 HDDs and slapping OMV on that. Or better yet, TrueNAS Core. But you said size is a concern and you want to use WiFi. I don’t know what to say, 4x 3.5" HDDs take quite a lot of space (about the size of an ITX board). And even on a Pi 4, I wouldn’t recommend more than 2 chunky spinning rust boys in RAID 1, due to the performance hit you get when running more than 3 drives on the poor thing. Unless performance isn’t an issue and just storage is, then 4 drives are ok (but I hate bottlenecks and inefficient use of resources, I love efficiency to a fault!). Maybe the RockPro64 is more capable with 4 drives (moar corez).

An integrated solution like Synology will be better looking and may or may not be more compact than doing it yourself (if you find a compact case meant for DIY NAS builds). But I’d still go with self-built stuff.

Also, just my $0.02, if you still want to stick with the Pi 3, get 2x 4/6 TB drives and RAID 1 them, instead of 4x 2/3 TB drives. As mentioned, even the Pi 4 struggles with 4 drives on USB 3. Oh, and USB 2 doesn’t support S.M.A.R.T. reporting (albeit smart is hit-or-miss on USB 3 as well, depending on your SATA to USB converter). Make sure you buy powered SATA to USB converters. Maybe look for something like this:

image
image600×600 47.4 KB

And 2 drives is literally half the space of 4 drives, so even better for size constraints. You can stick the Pi near the case (but you’d need quite some short cables). Or in absence of something like this and just having bare HDDs on the table, you can improvise a “case” (more like a holder) using nothing but some old hard plastic, some hot glue (or only hot glue) and lots of rubbery materials to dampen vibrations. I really love the old Dell laptop charger’s rubber belts. I don’t use it to hold HDDs like that, but I used it to hold my m.2 SSD metal case tightly near my Pi 4 metal case.

IMG_20210513_215520151
IMG_20210513_2155201514160×3120 3.02 MB

(you can’t really see it, it’s where the long USB cable is tucked, but these 2 metal boxes aren’t directly touching each other, to avoid too much heat spreading from the Pi 4 to the SSD - the rubber strap does it’s job - alternatively you could use cheap rubber wrist bands, they’re basically the same thing).

Now to ask questions: are you sure you want to use the Pi 3? Would it be cheaper to get some el-cheapo x86 components and slap them together than buying a USB dock? Can you find 2nd hand working USB docks for cheaper and go this route if budget is a concern? Do you want compactness, performance, price or a combination of them? You have to answer these for yourself :slight_smile:

2 Likes
17

I agree, even a cheap ($50*) used office PC would give better I/O performance, and still be pretty good on low power, but if OP already has a Pi3, it could be practice.

Then a dedicated box brought later, if/when more performance needed.

The USB dock I have, has a jmicron(?) controller, and I would not trust it for an array, and it looks identical to the one pictured, not sure if it is the same. It features UASP or whatever the high speed transfer is, so can go at drive speed, but it is the one that cuts out, dropping drives from the array, so beware?

*a cheap PC might, or might not have enough physical 3.5” spaces, and might, or might not have enough sata sockets/ sata power leads. If you pick it up at a recyclers, have a quick check inside, if you can

2 Likes
18

Thanks for the lengthy explaination, but I do still want to use my RPi3. As @Trooper_ish has mentioned I already have it lying around and it also partially serves as a practice.

In a year or so, I will know more about my requirements and I will also be able to build a more powerfull NAS that has everything I need/want. But currently, I don’t care that much about performance. (I mean sure, free performance is nice, but I also don’t mind if it lacks in that department.) At the moment, I care about size, price and compactness, all of which are offered by my RPi3. Naturally, I also would like my data to be safe, but it won’t be my only form of backup.

1 Like
19
  • ugh, btrfs with raid1 (or even raid1c3) for “special” and “metadata”, and raid5 for data in btrfs is ok… and you get per device checksumming this way.
  • Use spinning rust, stay away from USB thumb drives.
  • RPi4 doesn’t implement AES in hardware. If you want to encrypt the disks you may want to use adiantum, and you’ll want to be really careful about block sizes to minimize encryption overhead - devicemapper/luks/… might default to 512b instead of a far more reasonable 4k (disks these days typically do 512b emulation anyway and can write 4k blocks atomically, so cryptsetup/dm crypt at 4k should be just fine).
  • when attaching disks via usb3… be careful about the controller, not all controllers/chipsets support the UASP CPU efficiency feature, and some do it so badly that Linux detects them specifically and disables UASP for data safety reasons.
5 Likes
20

Why?

1 Like