Hi guys, just checking notes with the community, I just had one of my SSD mirrored pool die on me.
It looks like both SSDs failed at the same time, or they probaby failed over time, but hung on until I shut down and rebooted the server …
I had to do some electrical rewiring on my UPS line today, so I did a controlled shut down of my NAS (that failed) … unplugged all power from my UPS line and proceeded with the rewiring.
Once don I powered up again my NAS (an HP microserver Gen8, running 2 10TB HDDS, 2 60GB SSDs - mirrored root, 2 500GB SSDs - mirrored jails, vms and other stuff) … ony to discover my 500GB SSD pool was not coming up, and righty so, since the 500GB SSDs weren’t even initializing on the controller
I pulled the SSDs out and I can’t access either of them from my laptop with an USB to sata interface …
For now I just pluged in a spare 500GB SSD and restored from backup, and all is running smoothly so far (albeit on a single drive), waiting for a second SSD to come in to reestablish the mirror.
The SSDs are KingDian S280 480GB models , both bought from Amazon, one on september 2017 and the other on may 2018 …
It’s the first time in years I have an SSD fail on me, and never before had two fail in a span of weeks (last time the NAS was rebooted was probably a month ago) …
Just wondering how I could have noticed them failing … no SMART errors reported, no errors reported during scrub, no bad sectors reported … they just did not recover from a reboot …
Data in it isn’t really important, I recovered from a snapshot and I am only running jails and my docker VM, with no real data lost other than 24 hours of house sensors and prometheus monitoring … just wondering if anybody has a clue as to might have gone so horribly wrong and wat I could do in the fture to avoid the pain of restoring the whole pool from backup …
replying to myself just to document what I did to recover from the double SSD failure …
I had one spare 500GB ssd
booted the system with the new SSD inserted
deleted the disappeared pool from truenas gui, without selecting delete data/delete share/replication task config (this moved the system dataset to my other pool, and left the existing shares, iscsi config and replication tasks untouched, other than disabing them)
created a new pool with the same name, with a single vdev, unmirrored
restored the data in the pool from a remote snapshot on my backup NAS (yes, I have two, and yes, I replicate them daily):
set the restored snapshot to read write (when backed up it is automaticaly set to read only by truenas)
root@hpmicro01[~]# zfs set readonly=off ssdmicro01
rebooted, moved the system datased to the new pool, and re-enabled all dependent shares, then restarted my docker VM
*ordered a new ssd to match my spare one, once it arrived, I added it to my existing pool
root@hpmicro01[~]# zpool status ssdmicro01
pool: ssdmicro01
state: ONLINE
scan: resilvered 130G in 00:07:09 with 0 errors on Fri Dec 31 18:13:08 2021
config: