Zpool Offline. Any Hope of recovering anything?

[Exard3k]

@NicKF

Hopefully this thread acts as both a cautionary tale

Yeah, RAID is not a backup.

And ZFS shut down the pool to save your stuff after all. Being able to import in read-only was a very good sign. Thing can get way more complicated than this.

Were you able to identify why the entire vdev was ejected from the pool in the first place? I mean, two disks at the same time is unlikely a drive problem. And especially being the last vdev that was added. That smells fishy. And with all things being equal, it may happen again.

@NicKF

while he has gigabit download, he has only 30 Mbps upload, which is why I tried so hard to not rely on it).

Pretty much what my connection looks like too. The upload will do the job for sending incremental snapshots, but restoring an entire pool, yeah that’s probably weeks. I don’t have an off-site backup (I run with cold disks for replication). But I’d probably just load the server into my car and drive to my friend. Good opportunity for backup&beer party

I’m glad you got your stuff back. Having a storage array with disks failing beyond the level of redundancy…is is usually game over for most arrays.

[NicKF]

@jode

Depending on your snapshotting strategy you may be able to zfs send complete snapshots (that were created before you added the failed devices). That may have safed a lot of ISP bandwidth.

Also, use cp -a to keep file properties, in case you use SELinux cp -aZ.

Cheers!

Appreciate the feedback. You’re not wrong, it just didn’t really matter the ACLs were pretty basic. I’ve been just watching the CP run in top and tracking space utilization.

@Exard3k

Were you able to identify why the entire vdev was ejected from the pool in the first place? I mean, two disks at the same time is unlikely a drive problem. And especially being the last vdev that was added. That smells fishy. And with all things being equal, it may happen again.

It’s definitely something wrong with one of the controllers in one of my shelves. I’ve been writing an insnae amount of I/O using only a single path on the “B” controllers and everything’s fine.

@Exard3k

Pretty much what my connection looks like too. The upload will do the job for sending incremental snapshots, but restoring an entire pool, yeah that’s probably weeks. I don’t have an off-site backup (I run with cold disks for replication). But I’d probably just load the server into my car and drive to my friend. Good opportunity for backup&beer party

I’m glad you got your stuff back. Having a storage array with disks failing beyond the level of redundancy…is is usually game over for most arrays.

For sure, that would have been “easier” but a full size atx tower and 2 3U disks shelves aren’t a small thing to move around. We’re all here to learn from eachother also, so its all good.

[Exard3k]

@NicKF

It’s definitely something wrong with one of the controllers in one of my shelves

I’ll split my mirrors to different controllers. I couldn’t see any noticeable performance impact and if one controller dies or is “acting strangely” I got the other side of the mirror on the other controller and corresponding SFF-8643 connector, cable and backplane.
This obviously is more difficult with RAIDZ or if you don’t have multiple controllers.

[NicKF]

That’s how I designed it, but my action of “replacing” the failed disk after the system had automatically started to use my hotspare, coupled with the fact I didn’t properly document what drive was where, is what lead to the failure.,

[wendell]

Hey good work. There is a little known feature of zfs if you have a remote zpool you can actually pull down the missing bits from there so restoring just the damaged bits where local replicas are all gone is still recoverable! Glad this worked out though.