I know this was discussed elsewhere, but I want your opinions. I need to expand my TrueNAS ZFS Pool and I’m trying to decide between replacing all or some of the disks with larger disks, or adding another mirror.
Spare drive inventory is 2 x 4TB SAS Disks and 18 x 8TB SAS Disks
I have a 12 drive bays, and I’m only using 6.
My pool is made up of 3 x 4TB Mirrors. Should I just add the last 2 x 4TB disks I have in a new mirror, or start to replace the 4TB disks with 8TB disks?
Downside to adding another mirror is more power usage, and the fact when I do move to 8TB disks, I’ll have to remove a mirror. Any gochas?
Downsides of moving to 8TB disks is that it will take a bit longer to do, and has more risk
From my understanding of TrueNAS/ZFS, adding a mirror will only add redundancy, and not add capacity.
Since you are using only half of your drive bays, how about creating a similar setup using the 8TB drives? Then you can copy the data over to the new array, and never put data at risk, since the existing array is untouched.
Or, you can create a 6 drive Z2 array with the 8TB drives, giving you 32TB of space.
you can do either; however replacing existing drives in your mirror will mean you need to wait for each one to re-silver during which time you’re at risk of array failure. sort of - you’ll still have a backup as of the point you pulled the disk out that you’re replacing, as it was a mirror vdev.
adding a mirror will mean the data is unbalanced across the two mirrors. ZFS will write more to the new empty mirror until it balances out. performance won’t be as good as it could be, but if you’re only on gigabit ethernet then it doesn’t really matter.
I’ve done both. both scenarios work. Depending on how old your drives are, consider replacement to refresh the hardware, you’re going to be at risk of failure due to the oldest VDEV in the set, but be sure to buy CMR, raid friendly drives.
If you add more drives to a mirror - yes.
If you add another mirror VDEV to the pool, ZFS will stripe across both mirrors and you’ll have more capacity - essentially creating a RAID10 style stripe across two mirrors.
… and the fact when I do move to 8TB disks, I’ll have to remove a mirror.
You can’t remove a mirror vdev from a ZFS pool. (Well I mean you can now, but it doesn’t actually remove it. It makes a relocation map in RAM which isn’t something I’d want to do long term.)
FYI ZFS supports online replacement, so the new disk gets written too while the old disk is still in place, and the old disk is only removed after the resilvering has finished. Still slow but less risky.
And I think technically you can resilver a single disk from each vdev at once, with online replacement it doesn’t sound too crazy. But don’t quote me on that.
My standard process for the last 3 drive replacements I’ve done has been pop one drive out of the mirror, insert new drive, wait for re-silver.
Repeat drive by drive until the replacement is complete.
If you’re feeling particularly brave you could pop one drive out of each mirror two re-silver multiple mirrors at the same time, but you better get your drive identification right.
Hence, I just do one at a time, because I don’t trust myself and there’s no rush.
If you have a spare slot, you can insert the new drive and do “zpool replace old-drivenew-drive”, then the old drive stays in the pool until the resilvering finishes. This means your resilver can still read the old disk during the resilver, reducing risk.
Fit an additional 6x 8TB drives as a VDEV
Copy everything onto these.
Remove old drives.
Fit 6 more 8TB drives.
Merge these to the other VDEV to produce one huge storage area of 12 8TB drives.
