I currently have a TrueNAS server with the 3x 6-wide RADIZ2 data VDEVs using 24TB X24 Exos HDDs for a usable capacity of 288TB. The chasis I am using (45 Drives F8X2) has a capacity of 24 HDDs. I am wanting to add the remaining 6 HDDs to max this out, and I essentially have 2 options: Extend existing VDEVs from 6 → 8 drives OR add a 4th RAIDZ2 VDEV.
Simultaneous VDEV Extension
This is where my question comes into play: Can I extend all 3 VDEVs at once (one drive at a time per VDEV) to cut down the time this will take?
I have read several posts in this forum and others about extending VDEVs, but they all just talk about a single VDEV so I was curious if anyone has any experience with this. The obvious win with doing this is the final 6 drives would be 100% usable capacity vs only 66% if I add a new VDEV.
Other Hardware Specs
I figures there would be some questions on server specs, so here you go:
I am not a huge fan of expanding vdev, because I read to many issues online. On the other hand, just adding another vdev will suffer a similar “problem”.
Can you start over and backup the data?
I don’t know what your goal is.
You right now have 66.66% storage efficiency. Assuming you don’t have smaller files that suffer from pool geometry or padding. So maybe even less.
Also, are all drives from the same vendor? If not, I would go for 12x mirrors.
That gives you 16.66% less storage but way better IOPS and resilver times.
Or I would go with something that actually gives you storage. Something like a
2 times 12 wide RAIDZ2 (which would give you 83.33%, again assuming no pool geometry or padding issues).
Or even better draid2:21d:1s:24c or draid2:10d:1s:24c. That should give you 87.5% or 80% and faster rebuilds. But the minimum size file is either 84k or 40k. So I would add a special vdev SSD mirror. Which I probably would do for metadata performance anyway.
While I am not in a huge hurry technically, each extend will take a few days from what I can tell through research online. Given there are video editors using this during the day, I am hoping to start each extend session right at EOD on Friday so that there is as little impact as possible, maybe just a little time on Monday. If I can do all 3 vdevs at once this goes from a 6 weekend project to 2 weekends
I don’t have an on-site backup of this currently (work on progress), so restoring from a backup over the internet is really not ideal and would impact our video editing team.
The goal of this project is to maximize the usable storage capacity of the 24 drive slots, which in this case is doing 3x 8 wide RADIZ2 rather than 4x 6 wide RADIZ2. I mostly already have the performance I need so that is less of a concern
Yes, all of these drives are Exos x24 24TB drives, with the exception of a 24TB WD drive (RMA)
Unfortunately I don’t have the resources needed to do a pool layout change like you are suggesting, but either way, I for sure don’t want to do 12 way mirror as that would cut my storage efficiency way down.
As a note for metadata performance, I just ordered 3x 3.2TB SAS III SSDs for a 3-way mirror special vdev
I attempted to extend 2 VDEVs in the same pool at once so see what would happen, and the second extend job is hanging at 0%. My guess is that there is a lock or something the the first extend job put on the pool.