ZFS mirror question

Software & Operating Systems
1

Well this is my first post so let me start by saying “Hello all!” The question I have is I recently got 4 new WD Red 4TB drives (Made sure they were CMR drives) to upgrade a pair of mirrors in a zfs pool of 4 mirrors. I’m wondering the best way to go about this.

OS: Debian Buster (kernel 5.10)
CPU: Ryzen 5 3600 CPU
MB: ASRock Rack x470DU
The disks are all in hot-swap bays in a Norco 2U server chassis

This is the current setup:
root@ifrit:~# zpool status data
pool: data
state: ONLINE
scan: scrub repaired 0B in 00:08:32 with 0 errors on Sun Jun 13 00:32:34 2021
config:

	NAME                                            STATE     READ WRITE CKSUM
	data                                            ONLINE       0     0     0
	  mirror-0                                      ONLINE       0     0     0
	    ata-ST1000DM003-1ER162_W4Y6NJ9J             ONLINE       0     0     0
	    ata-ST1000DM003-1CH162_Z1DBCS60             ONLINE       0     0     0
	  mirror-1                                      ONLINE       0     0     0
	    ata-ST1000DM003-9YN162_S1D1QRM6             ONLINE       0     0     0
	    ata-ST1000DM003-1ER162_W4Y6NJ18             ONLINE       0     0     0
	  mirror-2                                      ONLINE       0     0     0
	    ata-Hitachi_HUA721010KLA330_GTF002PBJ481BF  ONLINE       0     0     0
	    ata-WDC_WD10EZEX-60M2NA0_WD-WCC3FPP9S2TY    ONLINE       0     0     0
	  mirror-3                                      ONLINE       0     0     0
	    ata-WDC_WD15EADS-00R6B0_WD-WCAVY0927527     ONLINE       0     0     0
	    ata-ST31500541AS_5XW0GCK5                   ONLINE       0     0     0
	logs	
	  mirror-8                                      ONLINE       0     0     0
	    ata-LITEON_CX1-JB512-HP_002547100MFL-part2  ONLINE       0     0     0
	    ata-LITEON_CX1-JB512-HP_002551102VTQ-part2  ONLINE       0     0     0
	cache
	  ata-LITEON_CX1-JB512-HP_002547100MFL-part1    ONLINE       0     0     0
	  ata-LITEON_CX1-JB512-HP_002551102VTQ-part1    ONLINE       0     0     0

errors: No known data errors

I’m looking to upgrade mirror-0 and mirror-1. Is it as painstaking as failing out a drive and replacing them one at a time? Or is there a means of just removing the mirror, swapping the drives, and adding the new mirror to the pool?

1 Like
2

Do you have any spare bays?
Or USB enclosures?

Or, spare headers inside the chassis?

The liteon drives, are they m.2?

3

Nah all 8 bays are full. The liteon drive are m.2. There are spare sata headers on the board though. The top on the chassis isn’t tool-less so I’d have to unscrew it and open it up and I’d like to avoid that it possible.

1 Like
4

Well, to be honest, the Quickest, and safest way, data integrity wise, would be to connect one of the new drives, and use the zpool attach, then zpool detach commands.

That way, all 8 data drives can stay connected, a 9th drive be zpool attach-ed to a drive in mirror. When resilvering complete, one drive can be zpool detach-ed, and swapped for another of the new drives.
Once the second new drive is resilvered, one could zpool detach the last old drive from that mirror, and swap the first new drive into a drive bay.

So at all times, there are unbroken mirrors.

there are other ways

2 Likes
5

Another way, would be to zpool detach one of the drives from one of the mirrors, swap the detached one for a new drive, zpool attach the new drive in to the single drive.
After resilver, you can of course do the same for the second drive in that mirror.

This can be, and is used, but it means the pool might (Might) die is there are some errors from the drive that is being resilvered. and is risky, as the whole pool is less resilient during the procedure

You could also export the entire pool, clone two of the drives to the new drives, and potentially re-import, with the clone in the bay of the cloned drive.

The old drive would still contain the partitioning it originally had, which should probably be cleared after the original pool comes up.

If you went this way, the I would say, do one drive at a time, and ensure the new drive imports fine, before doing the same to the other drive.

1 Like
6

@Trooper_ish first solution is the best you can do in this situation. It seems you used to add the drives by-id which means when you have detached an old drive from a mirror you can just pop the new drive in the hot swap bay and ZFS should do it’s magic and be able to find the drive in the new bay. I wanted to add, that before you attach the drives please run an extended SMART test as well as at least one pass of the badblockscommand to make sure the drives are in pristine condition.

2 Likes
7

Well looks like it’s a trip to Best Buy for a USB enclosure. I have a solid back up of the data in the pool, so that’s not the worry. Thanks for the help :fist_right: :fist_left:

2 Likes