I have 4 500GB hard drives which I am wanting to use in a ZFS array. I am wanting to have at least 1TB as the storage volume even if the storage size is 2TB. I think I know what I want to do but I wanted to drop in here and see what people think might be the best way to configure these. The requirements are that I have redundant drives and the pool won’t go down even if 1 drive fails. Whats most important is redundancy. What do you all think?
SERVER CPU AND MEMORY:
2X Intel X5660 CPU’s
24GB Ram
Start a single-parity RAIDZ (raidz) configuration at 3 disks (2+1)
Start a double-parity RAIDZ (raidz2) configuration at 5 disks (3+2)
Start a triple-parity RAIDZ (raidz3) configuration at 8 disks (5+3)
(N+P) with P = 1 (raidz), 2 (raidz2), or 3 (raidz3) and N equals 2, 4, or 8
Parity/redundancy increases as you go further up, storage capacity is inversely related.
Since you only have 4 drives then the best RAIDZ you can do is RAIDZ1, which will allow you to recover from a single drive failure, and use about ~75% of your total capacity. So you’d see storage in the neighborhood of 1.5 TiB.
Personally, I default to striped mirrors unless I have a compelling reason to do something else… that would satisfy your storage requirements and give you the best performance. It would also survive 1-2 drive failures.
I’d go for a mirrored stripe, aka raid10. If you use 4 drives for a raidz1 performance suffers, it’s best to use at least 5 drives for those. Speaking from experience here.
Edit: specified raidz1. Also, as @Dexter_Kane said, a striped mirror gives better performance.
A striped mirror would also allow you to grow the pool later on, expanding in size every other disk…
only needing to swap 2 disks for a larger pool rather than replacing them all before expanding
I made the error of using RaidZ1 initially and paid for it, thankfully loss was just my time in the test configuration I was experiementing with. I highly recommend you get the 5th disk and setup RaidZ2 at the very least. If you have an old SSD laying around it would be good to use as the ZIL drive (ZFS Intent Log) rather then having it on the primary array.
Technically, there isn’t a huge performance loss since raidz1 isn’t actually dedicating a single drive to parity, rather spreading the parity across all drives. However, it does allocate a single drive worth of space to parity. I’ve got an 80% full raidz2 array on 4TB HGST NAS drives without ZIL/L2ARC and I’m seeing 350MB/s writes into the array.
With 500G drives you will be fine with raidz1 and a spare if you need it. When you start using large drives like 4+G I prefer mirrors myself because rebuilding takes a long time of running the other disks 100% increasing the chance of fails.
And with ZFS if you make mirrored pairs of drives upgrading is easy just put in a larger drive in the pair. Rebuild then put a larger drive in the other drive in the pair and rebuild and you can jump sizes that way.
Sure it mean 2 drives to upgrade the raid size each time but it beats copying the whole raid off and back on to upgrade a raidz1 or 2
I think you missed the part where this is about ZFS, not general RAID. ZFS is far smarter about things then RAID and offers much better data integrity. It has it’s draw backs but personally I would chose ZFS RaidZ2 over a RAID10 array.
This is very similar to RAID10. You create a bunch of mirrored pairs, and then stripe data across those mirrors. Again, you get the added bonus of checksumming to prevent silent data corruption. This is the best performing RAID level for small random reads. How To Create Striped Mirrored Vdev Zpool
I should’ve specified how to do RAID10 in ZFS. I just expected people to realize you can do these types of things in ZFS just like any other RAID system.
I personally don’t like Parity RAID because it means putting more strain on the system’s CPU when doing writes due to calculating parity bits. ZFS already deals with checksums and other data integrity functions, so adding more to that is something I’d personally avoid. Especially if the server is doing anything else on top of running ZFS.
In OP’s post he placed emphasis on the best data integrity/redundancy, not performance. We know that RAID10 are good for performance, but what about integrity? Since he has four disks, how will redundancy hold up against other possible combinations with his amount of resources?
RAIDZ1 allows 1 drive to fail without losing data. If 2 drives fail, you lose data.
RAIDZ2 allows 2 drives to fail without losing data. If 3 drives fail, you lose data.
RAID 10 allows 1 drive to fail without losing data. If 2 drives fail, it depends which drives fail. If 3 drives fail, you lose data.
RAID 10 is a RAID 1 (mirror) of a RAID 0 (stripe). So you have two copies of the data (mirror) and those two copies are split between two drives (striped).
It’s easier to understand if you label them:
1A 1B
2A 2B
You need an A and a B to have your data be safe.
In this example, if you lose 1A and 1B, or 2A and 2B, your data is safe.
If you lose 1A and 2A, or 1B and 2B, you lose your data because half of the stripe is gone. Losing half of a mirror is ok. Losing half of a stripe is data loss.
It’s better than 1 drive redundancy but worse than 2 drive redundancy.
So let’s compare RAIDZ1, RAIDZ2, and RAID 10:
Name
Redundancy
Storage
Performance
Minimum Drives Required
Storage upgrade path
RAID 10
1 failure without losing data, 2 possible (see above example)
50% of original raw space
2x writes; 4x reads; no parity calculations
4
Upgrading Drive Size:
Progressively replace smaller drives with larger drives and wait for them to re-silver (mirror) before doing the next drive until all are upgraded. Upgrading number of Drives:
Stripe additional vdevs on newly added disks. This has to be pre-planned.
OR
The more complicated way to expand the stripes themselves is to drop one whole mirror, add larger or more drives, then re-silver (mirror) to the larger (in drives or capacity or both) mirror, then do the same thing with the original smaller stripe.
RAIDZ1
1 failure without losing data
~70% of original raw space
Less than 1x writes; depends on type of data for reads; CPU works on parity calculations
4
Upgrading Drive Size:
Progressively replace smaller drives with larger drives and wait for them to re-silver (mirror) before doing the next drive until all are upgraded. Upgrading number of Drives:
Stripe additional vdevs on newly added disks. This has to be pre-planned.
RAIDZ2
2 failures without losing data
~57% of original raw space
Less than 1x writes; depends on type of data for reads; CPU works on parity calculations
5
Progressively replace smaller drives with larger drives and wait for them to re-silver (mirror) before doing the next drive until all are upgraded.
Upgrading Drive Size:
Progressively replace smaller drives with larger drives and wait for them to re-silver (mirror) before doing the next drive until all are upgraded. You can do 2 at once, but that’s definitely not recommended. Upgrading number of Drives:
Stripe additional vdevs on newly added disks. This has to be pre-planned.
So you lose some storage space (about 20%), but gain a ton of performance for CPU and read/writes, while keeping “better than RAIDZ1 but worse than RAIDZ2 redundancy”. Also, you have a bit more complicated upgrade process for storage or what is the same process, but a fairly similar process for speed.
My perspective is that the odds of 2 drives failing is so low (presuming enterprise quality drives) that worrying about it is pointless, and even if it does happen, I now have a 50% chance to still keep my data.
Given that the OP has 500GB drives, they probably are old and aren’t Enterprise quality. But even then, two, at the same time, is still very unlikely.
This is where a lot of people get themselves into trouble. “Well obviously I want the most usable TB possible out of the disks I have, right?” Probably not. That big number might look sexy, but it’s liable to get you into a lot of trouble later.
…
Just like RAID10 has long been acknowledged the best performing conventional RAID topology, a pool of mirror vdevs is by far the best performing ZFS topology.
…
TL;DR
Too many words, mister sysadmin. What’s all this boil down to?
don’t be greedy. 50% storage efficiency is plenty.
for a given number of disks, a pool of mirrors will significantly outperform a RAIDZ stripe.
a degraded pool of mirrors will severely outperform a degraded RAIDZ stripe.
a degraded pool of mirrors will rebuild tremendously faster than a degraded RAIDZ stripe.
a pool of mirrors is easier to manage, maintain, live with, and upgrade than a RAIDZ stripe.
BACK. UP. YOUR POOL. REGULARLY. TAKE THIS SERIOUSLY.