I have an ongoing problem with disk space/volume growth. I’m adding 1-6TB every ~6 months and have been adding new disks at about the same frequency. So each time I’m down to 10% free, I buy a new disk (currently around 8TB is the sweet spot) and create a filesystem on it and then continue after doing a bunch of balancing. So at the moment I have 5x8TB disks, each with a separate filesystem and most are 80+% full, so I’m looking to add a new disk sometime in the next couple of months.
The problem is, manually balancing these volumes is becoming a pain. I want to add some sort of RAID to the mix, but can’t tolerate any data loss, so I’ve been just mirroring the disks to another physical server (i.e. This problem is the same on 2 machines) for hardware redundancy because I lost a PSU in the past and it took 2 drives with it.
So looking around the internet I see solutions of RAID 5,6,… and using LVM, BTRFS, ZFS, etc… but in all cases I see the same volume management problem that I already have when you need to add more physical storage.
So my question is: What’s the easiest/best option for ever growing large data size management that will properly protect the data from loss, without having a lot of volumes that will end up with different levels of protection from data loss? All of the solutions I’ve seen so far all have “growth problems” (i.e. Can’t just add a disk or two to the mix that doesn’t incur other major pains or usability problems… like I already have) that result in fragmented volumes or strange layouts that a loss of a disk results in catastrophic data losses.
Ideas? Articles? Suggestions??? Derogatory comments that I’m an idiot??
I think that btrfs is your best bet here. My understanding is btrfs has the capability to add different sized disks, one at a time.
When adding a single drive to, for example, a raid 5 or 6, the file system is stretched (I believe the term is rebalanced) to include the new drive without any sort of loss of redundancy.
No problem with ZFS. You plug in the drive and all datasets have more capacity. 3-4 clicks. All with redundancy, compression and all the ZFS magic. Only problem is you usually add 2 or more drives, depending on your configuration. But adding single disks isnt a thing if you want redundancy. But reading your posting, that isn’t an issue because you expect quite a bit of growth.
Another Pro is that you only have to deal with one filesystem covering all drives. It’s very easy to administrate.
With BTRFS you get into trouble with parity raid setups like RAID5 because it isn’t stable as of now and you still have to cover that write hole with UPS and a lot of faith.
Interesting, I’ve read a couple of articles that said that adding a new disk to a ZFS cluster of disks (using wrong terms on purpose here, to attempt to ensure we’re talking about the same concepts) will just add space as something like a separate volume without any RAID type of protection. My goal is to simplify filesystem management and add some sort of redundancy to the mix while keeping the ability to add a random size disk in the future without needing to rebalance/recreate any volumes/filesystems in the process (like I am now).
Were the other articles incorrect and there is a way to add a single disk to a cluster of disks to allow the volume to grow in size?
You may got something wrong there or we’re talking about different things. I think you are talking about adding a single disk as a vdev. Yeah that’s bad and you get red text and warnings for good reasons.
In ZFS you always add another vdev, which is a group of disks with their own redundancy. This can be a simple mirror of two disks or a RaidZ2 vdev with 10 disks. A single disk can’t provide it’s own redundancy (obviously).
In ZFS, you get two or more disks to expand the pool and also depends on what you already have/set up. That’s the caveat.
You may look into unRAID or Synology SHR Raid solutions for single disk expansions, but lacking featureset.
But if you expect constant growth, why not buy 2-3 disks at once and get everything you want, also considering redundancy reducing storage efficiency as well.
I was just going to comment this as well. I think there is a misunderstanding.
Adding 1 disk at a time does you no favors when you calculate/expect to grow every X months. Even a set of used drives added to the mix for a less expensive solution can help a ton. You’ve already done the projections and expectations…
I personally go with LVM because you can add a drive to the volume group if you have a Linear RAID, but if ZFS is on the table, it’s definitely the better solution. Also, and more importantly… How’s your backup scheme? Worried about losing a drive? Can you recover if you do?
My main concern is losing a drive. I have a PSU die and take 2 drives with it in the past, so I want redundancy on this machine and will be cloning to another physical machine for better resiliency. I have no way to backup such a large amount of information as my internet connection is limited to 50/10Mbps, and the cost of something like Backblaze would be too high anyway.
Buying multiple disks at a time isn’t really an option since 8TB disks go for ~$550 (my price after shipping) and my chassis only holds 10 drives anyway, so I’m attempting to keep the drive count low enough to be a reasonable price to maintain. I realize that I will have a problem sometime soon, but I can always add another chassis to hold more drives in a pinch.