Help understanding options for expanding ZFS storage server

Operating Systems & Open Source Linux
,
1

Hi Everyone,

I’ve been using a Ubuntu + ZFS storage server for the past ~6 years, and my current storage pool is starting to fill up so I am looking at expanding. Right now I have 4x4TB drives with RAIDZ1, and I am going to get another 4x4TB drives. I could use some help understanding some of the options for expanding my storage since I was getting a bit confused.

  1. I know I can add a new vdev to my existing storage pool, and if I understand correctly any new data will be striped across all vdevs? Meaning if my old vdev completely fails I would also loose all the data even on the new vdev?
  2. I could also make a whole new ZFS pool and keep the data entirely separate, with separate mount points on the system.
  3. I was also looking at making a new ZFS pool and then using mergerfs to combine them into a single filesystem.

I don’t particularly need extra performance from striping the data across drives, and I am concerned about the resiliency, with the older drives potentially causing data loss of everything. And I would like to keep everything as a single filesystem so that I don’t need to consciously route which ZFS pool different data is stored on to balance them. Those reasons lead me to that third point using mergerfs which is currently what I am leaning towards.

However, I don’t like having an intermediate layer, and I don’t know enough about ZFS to fully understand the options around expanding a ZFS pool, or if my concerns about adding a new vdev to the pool are justified.

tldr; What options are there around expanding my ZFS NAS server? What are the pros/cons of them? Am I being overly-paranoid avoiding adding a new vdev to my pool (because the 4TB drives should repair relatively quickly)? Is it common to combine something like mergerfs and ZFS to get isolated data but a unified filesystem, or is that a weird approach?

Thanks for all the help!

2

correct.

If one vdev fails, you loose your whole pool.

correct

I mean, yeah, but why? I would not do it, for the reasons you already mentioned.

RAIDZ1 was probably not the best decision in that case :wink:

Probably the best option would be to offload the data to your backup (you do have a offsite backup, right?) and after that rebuild your pool with a 8 drives wide RAIDZ2.

That will give you the same 75% storage efficiency (assuming you use larger files, no loss to padding or pool geometry) but now two drives can fail without you loosing the pool.

3 Likes
3

I essentially came here to say everything @ThisMightBeAFish said.

It is generally bad practice to layer file system tools on top of other file system tools, as they can wind up fighting with each other and having unintended consequences, up to and including data loss in some rare occasions.

I also agree that RAIDz was probably not the best choice, but for a four drive pool it becomes pretty difficult to stomach the capacity efficiency loss of the better alternatives (like striped mirrors or RAIDz2 or RAIDz3…

But there is one thing he didn’t touch upon.

Traditionally you had only two ways to expand an existing ZFS pool (without breaking it up and starting over). Those were:

  • Replace smaller drives with larger drives one by one, resilvering each, and once the last drive is resilvered if autoexpand=on is set, the pool grows to fill the space.

–OR–

  • Adding an additional VDEV that will then be striped, preferably of the same configuration as the first, but ZFS won’t stop you from mixing and matching. Mixing and matching VDEV configurations is generally not considered a good idea though.

(Worth nothing here that while striping for single drives - so called RAID 0 - has a bad reputation, and deservedly so, striping in this context, across stable redundant VDEV’s is well tested, and an intended way to use ZFS and as such is not a problem, as long as you plan your underlying redundancies correctly.)

There has been an update to this though.

Starting with OpenZFS 2.3 RAIDz expansion became possible.

You can now add drives to an existing RAIDz, RAIDz2 or RAIDz3 vdev.

–BUT–

When doing so, you cannot change the redundancy level. So, you cannot turn an existing four drive RAIDz1 vdev into a RAIDz2 or RAIDz3 vdev of any size. RAIDz expansion must retain the same level of redundancy as the original vdev being expanded.

What this means for you.

You can add a second vdev, and wind up with a pool consisting of two separate four drive RAIDz vdevs, but - honestly - in 2026, single drive parity is really no longer considered sufficient, especially as drive counts grow. You are essentially doubling your risk of total data loss if you do.

I know it is painful to hear, but the best path forward is to admit that you made a mistake with RAIDz1 back when you created the pool (no criticism intended here, we all learned and made mistakes at some point) and start over.

This means that - unless you want to buy all new drives, which is expensive, especially these days - you need a way to back up your data, while you break up your pool and re-form it.

Honestly though, you should already have this. Redundancy is not a backup, and you really ought to have backups of your data, especially if it is important and isn’t easily replaceable. Redundancy protects you against one or a couple of drive failures (depending on how you configured it) but it does not protect you against data loss due to accidental deletions, file system corruption, fire/flood/hurricane damage etc. etc. Definitely consider always having backups of all important and difficult to replace data.

But once you get over that hurdle, how should you set this up?

Back in the day performance was very sensitive to the combination of drive counts at each redundancy level based on partition level and stripe size as follows:

sub.mesa wrote:
As i understand, the performance issues with 4K disks isn’t just partition alignment, but also an issue with RAID-Z’s variable stripe size.
RAID-Z basically works to spread the 128KiB recordsizie upon on its data disks. That would lead to a formula like:
128KiB / (nr_of_drives – parity_drives) = maximum (default) variable stripe size
Let’s do some examples:
3-disk RAID-Z = 128KiB / 2 = 64KiB = good
4-disk RAID-Z = 128KiB / 3 = ~43KiB = BAD!
5-disk RAID-Z = 128KiB / 4 = 32KiB = good
9-disk RAID-Z = 128KiB / 8 = 16KiB = good
4-disk RAID-Z2 = 128KiB / 2 = 64KiB = good
5-disk RAID-Z2 = 128KiB / 3 = ~43KiB = BAD!
6-disk RAID-Z2 = 128KiB / 4 = 32KiB = good
10-disk RAID-Z2 = 128KiB / 8 = 16KiB = good

…but this is mostly irrelevant these days, at least from my understanding. At least unless you are chasing optimal IOPS, which for a file server you probably aren’t.

RAIDz just doesn’t cut it anymore for a very simple reason.

Lets say you have a pool of many drives and one fails. You now have no more redundancy. There is no parity to catch any potential silent corruption that may have occurred, and if another drive dies you lose the whole pool. And this is coming at a time when you are likely putting more stress on your pool than it has seen in a long time, by reading as fast as the system can support, in order to re-silver to the replacement drive.

Guess what. Aging drives + sudden high load = elevated risk of second drive failure, and if that happens all of your data is gone.

The goal these days is that you need to pick if you want to pursue performance/IOPS, or reliability/redundancy.

If you pursue performance, you have very regular backups you can restore from nearby, and you go for mirrors, as many mirrors as you need striped together. This is how you get the best ZFS performance possible for such things as databases, web hosting and other things which require high random access speeds / high IOPS / low latency.

Or you choose reliability / redundancy. This is better for lower intensity file storage. You can make better use of your storage (you don’t have to dedicate at least half of it to redundancy like you do with a mirror) and you gain more drive failure tolerance (if you go above RAIDz) at the cost of a performance hit.

Generally people who know lots about ZFS tend to recommend RAIDz2 VDEV’s in the 6-8 drive size these days, and if that is insufficient, adding a second RAIDz2 VDEV. This way you have some redundancy in place while you are resilvering, and thus you have a lower risk of data loss.

You should still back up your data though. Redundancy is never a replacement for backups.

For some very specific applications (like slow remote backup servers limited in regular operation by WAN speeds) some people will go up to a slightly higher drive count and use RAIDz3, to be able to absorb more simultaneous drive losses, but the downside here is that resilvers - when necessary - will be much slower on a RAIDz3 VDEV than on RAIDz2. Sometimes they can take days.

One thing you can try to do to ease the pain of the initial data transfer, is to create already degraded pools, transfer your data to them, and then swap in the missing drives to create a new vdev in a different size and with different redundancy, but this is highly risky, and really not recommended unless you know what you are doing and understand the risks.

I hope this was helpful.

4 Likes
4

Keep in mind this is from some one who just tinkers, but has made plenty of mistakes and attempts to learn from them.

How much physical space do you have along with room in your server. Getting more and more smaller drives might not be the best long term approach (stares at a stack of 3tb SAS drives on the work bench)

What is your budget? Space in your homelab/server? How much electricity do you want to use? I think a question worth asking is if you can, get 3x 8tb drives. End up with more storage in less drives, and if prices go back down, snagg another 8tb and expand the pool for even more space.

In my mind if you get more 4tb drives, you are committing to running many more drives for the foreseeable future.

3 Likes
5

That is a good point.

Normally one solves this by just swapping out the drives one by one for larger ones, and then having autoexpand grow the pool…

But, OP’s best choice (which helps solve all three problems he has, too low redundancy, space running out, and the issue of what to do with his data during the transfer) might just be to get six larger drives, set up a new RAIDz2 pool with those six drives, use zfs send/recv to transfer the data from the existing pool to the new one, and then move the 4 drives to a different application (maybe a backup system for important data?)

The backup system would be smaller than the primary one, but that is often the case, and usually isn’t a problem, because usually critical/important/non-replaceable data that is worthy of a backup is a smaller subset of the total amount of data on a pool.

2 Likes
6

That was my same thought on using the existing drives for a backup.

I also think that small drives only have value right now because everything is so expensive.

If things “normalize” smaller drives will probably turn into ewaste. And you are stuck running a whole bunch of them to keep your pool going. Want to upgrade? That is a bunch of drives to swap out unless you want to rebuild the array?

How many of us have (or had because we got rid of them) stacks of 500GB or less.

7

I think you are right.

I have 12x 16TB Seagate Exos x18 drives in my main NAS system right now which are getting up there in age… Oldest one has 40910 hours on it. Youngest (an early RMA replacement) has 20639 hours on it.

I bought these brand new back in 2022 at $279.99 a piece. If I have to replace them at current prices I’ll be in trouble… Even used 16TB enterprise drives are selling for over $500 now :grimacing:

If I start having failures I will probably be forced to plug the gap by hunting for used drives and hope the market returns to some assemblance of normality sooner rather than later so I can replace them all before they enter truly geriatric territory.

2 Likes
8

ThisMightBeAFish Thanks for answering so many of my questions, that was really helpful! I have a backup of important data but lots of painful to loose data (e.g. Blender renders that take ~1week to render but I have the original scene on an offsite backup so I can recover if absolutely necessary, it’s just painful/slow).

@mattlach Thank you for your really detailed write up, that is very helpful, and makes a lot of sense.

I was worried about something along those lines, so thanks for confirming/explaining it.

@unrumple Yeah, with the absurd price of everything, this expansion is entirely a stop-gap measure. For my usage the ~11TB for the 4x 4TB will last me ~5 years and I’m hoping prices are back to reasonable amounts by then. But you’ve got a great point about the smaller drives being ewaste after that.

Budget wise I’m trying to keep the upgrade <1k USD, but I’m flexible on that and the 4TB drives I found were $100 each, and the only 8TB drives I found from a quick search (used or new) were ~$300 each :grimacing:.

Thank you all for the explanations and input, it’s been genuinely super helpful.

It seems the recommendation, if using the 4TB drives, is to rebuild the array as 8x 4TB RAIDz2, as much as I’d love to be using larger capacity drives the pricing at the moment is just a bit too much (I’m kicking myself I didn’t do this expansion a year or two ago).

So I’ll make a full off-site backup, rebuild the array using RAIDz2, and pull the data back down to the new larger pool. Then once/if prices “normalize” I’ll go with mattlach suggestion to get six larger drives and transfer the data to that new pool, and move the older pool to a separate backup system.

One again thank you all for the input it’s been incredibly helpful!

3 Likes
9

Yes and no. It has nothing to do with “these days” but about the size of your stripes.
VM that use zvols use 16k by default. So ALL the data of that VM disk will be multiple 16k writes. Here the geometry and padding issues kicks in extremly hard.

Datasets on the other hand use 128k by default. 128k is so big, the padding or geometry issues becomes less relevant. But that 128k is just a max value. It can be any multiple of 4k. So If you only put 4k files into your 128k record size dataset, every files will become a 4k record and suffer extremly from these issues.

Link to efficiency table and explanation

128k is missing there, so we will do it for ourselves for a 128k record size, 8 wide, RAIDZ2:

RAIDZ2 with 8 drives

With 8 drives, we get a stripe 8 drives wide.

But not all stripes have six 4k data sectors (24k) and two 4k parity sectors.
For a volblock of 128k, we need 5.33 stripes (128k/24k).
Five stripes have six 4k data sectors, which offers in total 120k data. 
These five stripes each have two 4k sectors for parity, in total 40k.
The sixth stripe is smaller, needs only two 4k data sectors for the missing 8k (128 - 120).
The sixth stripe also has two 4k sectors for parity.

In total, we have 32 4k data sectors and 12 4k parity sectors.
So 32 + 12 = 44 sectors. 

44 sectors can't be divided by 3 so there is padding needed.
We add one 4k padding sector. That gets us to 45k sectors. 45 can be devided by 3. Hurray, we did it.

So what do we get?
We now have a 180k stripe to store 128k of actual data. 

We expected a storage efficiency of 6/8*100= 75%, but only got 128/180*100= 71.11%!

Again, remember that this only applies to 128k records! But it does not matter how large the files are, with 128 record size you can't go beyond 71.11%.

You can of course set the record size to 1M to once again make this issue smaller. 
In that case you should be carful not to put stuff like DB on it, which would suffer from bad write amplification when only doing 16k writes. 

So the quetion is OP, what do you plan on to store? Huge files for Plex?

TLDR: If you only have big files, no blockstorage for VMs, use the default recordsize of 128k, for a 8 wide RAIDZ2, you won’t get the expexted 75% storage efficiency, but only 71.11%.

10

That’s helpful to have the additional context, and thank you for the detailed explanation.

I have a separate SSD mirror for for smaller / more frequently accessed data like DBs or logs. So this HDD pool is primarily larger files, I checked the average file size of my current data is ~200MB although there are quite a few smaller files as well (metadata or photos), but I can move some of those to the SSDs instead.

But it’s very helpful to understand about that so I’m not surprised when it’s less efficient that I expected.

1 Like
11

The default size for those in the datasets was changed from 8k or 16k to 128k at some point in the ZoL project.

So this is what he meant with “these days”.

12

nope, you are mixing up stuff.

Default size for zvol was 8k, later changed to 16k.

Default size for record size is and was always 128k.

2 Likes