2-drive server setup suggestions

[m4iler]

Hello everyone! I come to you for advice (what else).

I recently (2 hours ago) bought a 4-bay dock and a 4TB WD Red. I already have a 4TB WD Red and would like to make the whole setup more reliable/resilient to drive failure.

As it stands now, I have it set up as follows:

• Optiplex 7050 micro running Proxmox and several VMs
• 500GB NVMe drive for OSs
• 4TB drive connected via USB to the Optiplex

With the new setup, the only thing to really change will be a 4-bay dock instead of a single USB harddrive box. Now I have to think about the future development of this setup.

My thinking is to, over time, put in 4 of the drives and then switch them out with bigger ones (spending 200$ every year is better than spending 1000$ all at once). However, my issue is how I’ll set up the filesystem. I’m thinking about a ZFS mirror, or some type that will give me resilience (if one drive fails, another will be able to read), and some self-repair.

Is there a good enough setup for 2 drives that would give me this sort of resilience? Or is all the good stuff meant for 3+ drives? What would you recommend for this setup?

Thank you!

[exovert]

Expanding pools with zfs, now isn’t that plug and play but there are means.

https://www.youtube.com/watch?v=11bWnvCwTOU

You can start with a 4+4 mirror, add another 4+4 mirror (and then you have a pool of 2 mirrors). (be sure to listen to the ‘one final note’ from 16:02)

I don’t know much guidance the proxmox interface will offer with this so it might have to be done at command line.

if you don’t already, i’d also somehow to try to borrow another drive to use as backup before doing any operations on the pool structure.

I’ve used a 4 bay usb(/firewire) dock - the cables are not cat proof (and speed may not be that great, either). I’d recommend to be sure you use your highest rated usb port on the optiplex (you’ll find some of them are usb 2.0 only) and i’d try to cable tie it, to something so it can’t be as easily disturbed.

raidz expansion meaning adding a disk to a raidz is actually in testing, but is not in the public release yet (and will likely be longer before it gets into proxmox).

https://github.com/openzfs/zfs/commit/5caeef02fa531

a raidz is also going to require 3 disks to setup, and just increasing one disk at a time isn’t going to give you space.

best of luck

[quilt]

I’m confused, you have 4 drives or 2?

Consider btrfs. The drawback is that raid5/6 is broken but it is very flexible with raid 0/1/10. You can have a pool with different sized drives, flexibly add/remove drives etc. In this case I‘d go btrfs raid 10 with 4 drives, personally.

Edit: I guess you are starting with 2 drives and are adding over time to 4, and then will replace with bigger drives? In that case I‘d still opt for btrfs. RAID 10 should work with 2, or 3 drives too on btrfs.

[ThatGuyB]

Let me understand this properly. You have a 4-bay dock, but you can only afford 2 drives for now. That’s fine.

With 2 drives, you can only go with a mirror, which you might be surprised how much it can hold its weight if you’re just one person (and even with more people, mirrors are still fine depending on the size and workload - I’ve had 70 people edit documents and copy files from 2 mirrors, different pools, although I wasn’t the one to set it up, that was dumb).

@m4iler

over time, put in 4 of the drives and then switch them out with bigger ones

You can expand a 2 drive mirror by adding 2 more drives in a different mirror, and stripping the 2 mirrors (RAID10 basically). Later, you can replace 1 drive at a time and the pool will still be the same size (of the smallest disk). When all drives are replaced with higher capacity ones, the pool will expand in size to the now smallest vdev in it.

Since you are doing the cash-flow approach and not the future-proof investment (and even if you didn’t, still a good thing to do), you should create partitions on the drives and add the partitions to the pool, like say /dev/disk/by-id/disk-part1, as opposed to giving the whole disk. That is because disk sizes might differ when you get the newer models (or completely different skews or even brands).

@m4iler

Is there a good enough setup for 2 drives that would give me this sort of resilience?

Mirror.

@m4iler

Or is all the good stuff meant for 3+ drives?

Mirrors and stripped mirrors have the same data checking in ZFS. Also, raid-z{1,2,3} are not (easily) expandable (except for stripping them, which takes a lot of drives).

@m4iler

What would you recommend for this setup?

2-way mirror, upgrade to stripped mirrors later by adding 2 drives.

Personally, I’d recommend you have a backup server with a decent capacity (to allow you to keep more than just one copy of your data, to hold some older backups, just in case). And when time comes to upgrade, just export the pool out of your main system, take out the drives, create a new pool and zfs-send from the backup server to the new pool.

Way easier and faster than having to resilver a drive at a time and less risky (you’ll still have your other disks you took out if your backup server somehow has both drives fail right when doing that initial zfs-send to the new pool). A bit more pricey, but a backup server can be something as cheap as an Odroid HC4 (that’s what I’m running), allowing you to invest more in the drives (for the bigger capacity).

If these odds weren’t low enough, then even lower are the odds that your old pool will also fail when you run another full backup (zfs-send) to a new backup server, in case your old backup pool failed. Maybe if you lived in a van and your hardware experienced daily “earth-quakes,” maybe the scenario wouldn’t be as improbable, but if you live in a normal house, the odds are nearing zero, ceteris paribus.

So, I’d go with 2 drives now, then a backup server with a higher capacity than the total of the future stripped-mirrors pool, then 2 more drives for the fresh stripped-mirrors pool, then just upgrade the entire stripped-mirrors pool (all 4 drives at once, so you don’t lose the performance by going from smaller stripped-mirror to larger mirror only).

[sephkeene]

Two WD Reds? Are they Conventional MAGNETIC Recording drives (CMR) or Shingle Magnetic Recording drives?

I ask because SMR drives should generally be steered away from for NAS use. They can take a long time to perform sustained writes, such as resilvering a drive.

I the listing of the seller where you bought the drive it should say SMR or CMR.

[m4iler]

One is for sure a WD red plus, one is a WD red normal. I assume all of them would be CMR.

[m4iler]

Yeah, I messed up my explanation

I have:

1 4TB drive (currently in use, half-full)
1 4TB drive (new, not in use)
1 DAS device with 4 drive slots

I have to do the following:

• backup the important stuff (I have a 2TB drive that I can use to store most of it, the rest will go into random places)
• Think of a storage strategy for 2 drives so 1 can die (now btrfs sounds nice, too!)
• set it up, move storage back over

In the future, I’m hoping to buy two more drives, add them to the dock and set them up as another mirrored setup. Basically, it will be 2 drives in RAID 1 + 2 drives in RAID 1. 50% capacity, but any 1 drive can die. Alternatively, if I could move it to a RAID10 when I have 4 drives populated, that would be great.

My main thing I’d like to do is to upgrade the drives one by one. Buy a drive, put it in one of the mirrors instead of the old one, and then buy the same drive later and add it to the same mirror. Alernatively alternatively, I could buy pairs again and do a shift-around:

• mirror1 is to be upgraded
• mirror2 is not to be upgraded
• I take out mirror2, replace the drives with larger drives
• move data from mirror1 to new drives
• take out mirror1 drives
• put in mirror2 drives (those didn’t change, they can still work)

Since I don’t have too many drive bays, that’s the only way I can see of this working out.

I’ve matured enough to come to terms with 50% capacity (in the past, I wanted JBOD), so I’m asking around for ways people have this set up

[m4iler]

My main issue will be with the “backup” server. That will probably have to be taken care of with a RasPi or something with one drive (the first to go out of the server).

I don’t want to jinx myself, but I ran with 1 drive for years and only lost stuff when I rm’d the wrong thing. I’m sure a remote backup will be necessary in the future, but I’m taking baby steps due to other expenses. As it stands now, I have 2 drives, one of which is half-full, and need a way forward that will be future-proof in terms of storage upgrades: If I buy 1 or 2 new drives, I want to be able to add them to the current storage space with as little tomfoolery as possible. I understand that will not be easy, but there must be something close to optimal.