What raid would you recommend for 4 drive config? [x4 28TB]

I have a DXP4800 Plus from UGREEN and have been really enjoying it. I previously had two 10TB drives mirrored but maxed that out because I host Jellyfin for my friends and family and they watch a lot of content.

I finally pulled the trigger and bought 4 28TB drives that will be arriving tomorrow and am trying to do research whether it should be in a striped mirror or RAIDZ2.

I also plan on using two 2TB 990 Pro’s (upgraded to the latest firmware) as a special VDEV for metadata and small block storage.

Looks like in openZFS 2.4 it will also be used for the ZIL which is nice because my homelab is currently using a NFSv4.2 Kerb setup.

What do you guys think?
Mirror + Stripe? - Performance but less redundancy
RaidZ2? - Redundant but less performance (I read that resilvering large drives takes FOREVER, but I don’t have experiencing in it)

Also yes - raid is not a backup, I won’t be backing up my whole pool as the movies/shows I can re-download. I typically reserve 1-2TB for personal data such as immich photos, etc which will be backed up.

Use RAID5: with 4 drives you get 3x capacity of the smallest drive and 1 drive redundancy. NEVER use a RAID version with a 0 (zero) in it, you’re just waiting for your data to go poof :dash:

But I doubt the ZFS crowd even gives you the option to not-ZFS your filesystem :roll_eyes:

Depends on how important the data is to you.

RaidZ1 (software ZFS with one redundant disk) is the way to go in this modern day and age. I also recommend you keep an eye out for SSDs, as those are going to start coming down in price significantly once the latest AI craze is coming down. A single 122TB drive could go as low as $1000 by 2028, that is less than $10 per TB.

Depends on the redundancy vs. capacity target you have.

If you want more redundancy I’d say RAID6 (two drives fault tolerance, less write speed) or RAID10 (two drive fault tolerance on different mirrors, double the write speed).
If you need capacity go for RAID5.

Since you only have two special vdev drives that would mean they will be mirrored, so it doesnt make sense to put the underlying drives in a raid6 type array to me, if you even can do that? Im not sure of the limitations on metadata drives with the configuration of the storage drive parts. Maybe you can maybe you cant, but either way when the metadata drives are a simple mirror it just seems like the storage drives should also be a mirror since having a second drive redundancy at the storage level is somewhat negated by the single drive redundancy at the metadata level. Remember that if you lose the special vdev then all your data is gone, just like if you lost your storage drives. So if you feel the need for a certain level of redundancy on your storage you should feel the same level to your special vdev.

Classic topic! I wrestled with this when starting a new Truenas build.

TLDR;
Mirrors are better than RAIDZ2 for 4 HDDs in ZFS

Four drives is a tricky size, and not generally one I would use with any kind of RAID other than a dual mirror, and even then I’d only use a dual mirror for performance critical applications where data integrity is less important (and honestly, for anything performance critical, I wouldn’t use a hard drive anyway)

Dual mirrors are not very space efficient though. If you want more capacity you can go with RAID5/RAIDz1 but that has generally not been recommended for quite some time now.

The 2007 article “Why RAID5 Stops working in 2009” was quite controversial at the time, but over time just about everything in it has been accepted as true by those in the know.

The primary reason is this:

As drives have become larger and larger, they have a frighteningly large number of bits. Each of those 28TB drives of yours have 2.24*10^14 bits.

If you ever have to resilver that pool, and it is close to being full, the resilver process will be reading three times that (the three remaining good drives) so, 6.72*10^14 bits.

Depending on your drive you will likely have a claimed URE (unreadable error rate) of either 10^14 or 10^15.

If you only have one redundant drive, if one drive fails or needs to be replaced, you are reading from the rest during your rebuild without any redundancy at all. In other words if one bit is either unreadable or flips, you have at least one bad file.

If your drives are rated at a URE of 10^14 this will happen 6.72 times during the average resilver for you. If your drives are rated at 10^15, it will happen an average of 0.672 times.

Ideally for data integrity, if the data matters to you, you want at least two redundant drives, such that you still have redundancy during a resilver, and replace any failed drive immediately as soon as it fails before another one does, so you won’t suffer from URE’s or flipped bits during that resilver.

It seems like you have already bought the drives, but honestly, if you had reached out before doing so, I would probably have recommended getting more smaller drives rather than four 28TB drives. A six drive RAID6/RAIDz2 pool would be perfect for this application and to get the same amount of available space as a dual mirror with the drives you did buy could have been done with six 14TB drives. Getting the same available space as a RAID5/RAIDz pool with the drives you did buy could have been achieved with six 21TB drives. This would have been better for data integrity long term.

That said, RAID should not be mistaken for a backup. It can help you with uptime (not having to restore from backup) but in and of itself it is not backup. If the data matters to you, you should always have a backup (preferably offsite) in addition to raid.

With something like Jellyfin - however - you might consider any data on the pool to be replaceable, in which case maybe it doesn’t matter.

If it is a mixed pool with some replaceable data (media files) and some data that is irreplaceable (family photos, personal files, etc.) I’d highly recommend keeping a regular offsite backup of the irreplaceable data.

RAID is great, but it only protects you against drive failure. It doesn’t protect you against lightning strikes, fires or floods that take out the whole system, pool corruption (which is granted pretty unlikely with ZFS, but has known to happen), errant keystrokes or the likes of malware/viruses/ransomware.

For these you need a true copy somewhere else you can restore from. All RAID should ever really be is for convenience and timeliness, to reduce the risk of you having to actually restore from backup. It should never be your only protective measure for critical data.

This article was written back in the day when HDDs were using archaic RS code for their ECC and was probably colored by that technical deficiency (yes I’m calling Reed Solomon a technical deficiency in this context).
Ever since we went to LDPC+soft-decoding, HDDs have far exceed their spec sheet’s quoted URE rates if run in reasonable conditions.

As a data point, the current stats on my raid 60 array are:

59.786732063539 Read Gigabytes Processed[10^15 bytes]
605209 Read Errors Corrected By ECC Delayed
0 Total Uncorrected Read Errors

EDIT:
Those error stats are of the constituent drives themselves, not after the raid 60 resolving an error (because it’s never had to).

1 Like

raid 6