So I’ve used FreeNAS for a while and I need a few simple questions answered.
I have always increased pool size by adding an additional VDEV of identical size and layout as the previous one. Meaning if I have a VDEV consisting of 8 x 8TB Drives in RaidZ2… To increase the size of the pool, I have created and added an indentical VDEV consisting of the same 8 x 8TB Drives in RaidZ2.
Now to my simple question… Can I add a new VDEV to the same existing pool with the same layout but larger drives. For example, 8 x 12TB also in RaidZ2?
Follow -up simple question… Will it be able to use the full size of those new 12TB drives or will it be reduced to only 8TB of the 12TB drives until the entire pool becomes 12 TB disks (Like if I did replace on them)?
Slightly more advanced question… Assuming that I can add a new VDEV with larger disks and it will see all of the space, are there any major performance issues with having a pool that consists of different size VDEVs?
I feel like I should know the answers to these questions and I have read up a ton on FreeNAS / TrueNAS… But I feel like I have never really seen a definitive answer on this exact scenario.
The short answer is yes, you can but there certain things you should now before expanding your pool.
The capacity of a pool is dictated by the usable cacpity of the vdev, or vdevs that it contains. The capacity of each vdev is that of it’s smallest disk. For example, if you were to do something really crazy like adding a single disk to your pool. That single disk would be it’s own vdev. The capacity of the single disk vdev would be the capacity of the smallest disk times the number of disks in the vdev, minus any parity or mirroring, minus overhead, and possible minus swap space (if you’re using a swap partition). In other words, a vdev consisting of a single 1TB drive, would have a capacity of 1TB.
The same formula is true of other vdev types. A mirror of 2x 1TB disks would be capacity of the smallest disk times the number of disks in the vdev, minus any parity or mirroring (so minus 1TB), minus overhead, and possible minus swap space.
In the case of a RAIDz* vdev, it’d be the capacity of the smallest disk (let’s say 7x 1TB and 1x 8TB), minus any parity or mirroring (let’s say were using RAIDz2, so minus 2x 1TB), minus overhead, and possible minus swap space. So we end up with 6TB or there abouts.
The capacity of a vdev is a multiple of it’s smallest disk, whether that’s 1x, 2x, or 20x, regardless of whether we’re talking about a single disk vdev, a mirror, or RAIDz1, 2, or 3.
The usable capacity of one vdev has no effect on the usable capacity of any other vdev.
As I said at the beginning, the capacity of your pool is a sum of the usable capacities of all it’s vdevs.
So in you scenario, according to the default settings on https://wintelguy.com/zfs-calc.pl, your pool would have a usable capacity of 98.84 TB or to use the TrueNAS standard of Tebibytes… 89.89 TiB… give or take (as TrueNAS allocated 2GB of each disk to swap space).
This is moving a little bit out of my comfort zone, so I’ll leave someone else to provide a more detailed, informed answer but what I will say is that the parity overhead… how complicated the parity calculations are, can result in a performance impact. Which is one of the reasons why it is advised that pools are expanded using vdevs of the same topography and type as the exisitng vdev(s) within the pool. So you should stick to adding RAIDz2 vdevs to a pool that already has a RAIDz2 vdev within it.
The other main reason for sticking to the same vdev type is that your pool is only as resistant to failure as it’s least resiliant vdev. RAIDz2 or even RAIDz3 count for nothing if you were to add a single disk to your pool, which then fails, taking your pool with it.
Whether having a different number of disks within different the vdevs, which can add complexity to the parity calculations, would result in a “major” performance issue, I can’t really say (as my primary concern has always been capacity rather than performance) but perhaps someone else can post more concrete numbers in regards to that. The speed of the disks can also effect performance, so try to match the RPMs of your spinning rust, don’t mix SSDs and HDDs in the same pool etc.
As I said though, this is moving out of my compfort zone, say take this part of my post with a grain of salt.
I know what you mean. I struggle to learn from purely text based resources. I find myself going round in circles and tying myself up in mental knots, and if you’re anything like me, when I’m trying to answer a specific question, I need a specific answer. Just having more general knowledge on the subject and thus having the tools to come up with the answer myself doesn’t always help. That’s where helpful, friendly communities come in. So feel free to ask any questions you want.
Here’s are a few great ZFS resources that might help you.
There’s also some fantastic videos on youtube, that I’ll link to as soon as I get the chance.
It’s also worth checking out the official FreeNAS… damn, I mean TrueNAS forums as they’re a much more welcoming place than their outdated reputation would have you believe. And as much as I loathe the site itself, the FreeNAS subreddit, which is frequented by the devs, can be very helpful and welcoming too.
the tldr is that if you want to grow your pool piecemeal (sans rebalancing) then you must use mirrors because if you use RAIDZx then you will have to incrementally upgrade all the VDEV’s before the new capacity is recognized.