Successful QNAP Data Recovery

(I debated on reviving another older thread on QNAP data, decided against it because that thread seems to have died in 2018.)

Friday night; found that my TS-253B was no longer responding on the network. We’ve had some power outages recently and I assumed I had intentionally powered it off. Went to the cabinet, saw it was on. I force-powered it off and as soon as it went off the status light came back on red and started flashing.

This wasn’t a problem I felt like dealing with on a Friday. I’ll just go get the file I needed from my backups. Oh good I never finished getting those set up for some reason.

(I’m going to shoutout @jrcichra here at the beginning for dealing with my incessant questions through this)

Step 1 on the road to recovery was just to get the drives copied so we can try to recovery from copies without risking the drives. I had some issues with my SATA docking station playing nicely on my server, so I just pushed the data from another machine on the network. Annoying, but doable. I’m not sure why exactly, but once the read side dd had completed, it didn’t close, so the write side also hung open. Once they stopped transferred, I just terminated them. It didn’t’ cause me any problems, I just thought I’d mention it.

Server Side:
nc -l 21001 | lz4 --decompress dd of=/z4x14/qnap_recovery/WD40EFRX-68WT0N0.img bs=4k status=progress
Workstation Side:
dd if=/dev/sde status=progress bs=4k | lz4 --compress | nc 10.0.0.252 21001

Step 2 was to validate the drives and images had the same data. md5sum to the rescue. I am not hugely confident in using the the md5sum command to automatically compare the hashes, but I know how to write them to a file. This also gave me the opportunity to see how far along the drive those were.

Server Side:
pv /z4x14/qnap_recovery/WD40EFRX-68WT0N0.img | md5sum > /z4x14/qnap_recovery/WD40EFRX-68WT0N0.img.md5sum
Workstation Side:
pv /dev/sde | md5sum > WD40EFRX-68WT0N0.disk.md5sum

Output from those files. I know there’s better ways to do it.

WD40EFRX-68N32N0.disk.md5sum:fda5493b4dbbb784e459c75ff499a7ce -
WD40EFRX-68N32N0.img.md5sum:fda5493b4dbbb784e459c75ff499a7ce -
WD40EFRX-68WT0N0.disk.md5sum:dc76f7c11fb29a7a2a60bf1445efd9ab -
WD40EFRX-68WT0N0.img.md5sum:dc76f7c11fb29a7a2a60bf1445efd9ab -

Step 3 was to spin up a VM and try an seemingly endless list of things. It seems like it should have been just mdadm --assemble and then mount the ext4 filesystem that is presented. mdadm does present a drive; and you can see it. but then there’s a drbd volume on there which even once we got a working drbd config wouldn’t read correctly. Tried using binwalk to find the filesystems and then mount -o offset=nnn but that gave an error about the filesystem needing cleaned.

One of the posts in This QNAP Forum Post lead me to basically believing this was never going to be possible.

…given that QNAP maintains their own fork of the Linux Kernel…

That got me back around to thinking that the nas is just an computer. It runs some software, surely we can get that virtualized. It was getting kind of late, but I pulled the TS-253B firmware from a QNAP post detailing how to recover from a failed firmware update. Messed with it for a couple hours and went to bed.

Step 4 Monday morning I remembered that QNAP offers their software as a “run on your own hardware” appliance. $5/month for a single core license. QuTScloud for private cloud if you. (I tried to throw in a link, it seems unhappy)

I figured that was well worth the price of admission and figured it was worth a shot.

Created a new VM without attaching the drives, did a basic install. (It did require me to attach a drive with at least 200GB of free space.) Once I was able to get logged into the normal web console and saw it was pretty much exactly what the physical nas had, I powered the VM off, attached the drives and was able to recover the storage pool in the VM.

From that point, It was just a matter of downloading the data.

Check on your backups before you need them.

I hope this helps somebody in the future, I was really stressed out about this.

2 Likes

A rewarding story. It may give hope to someone sometime.

If it hasn’t been checked it only might be a backup.

2 Likes

We need a world restore day at least as much as we need a world backup day

2 Likes

Glad you got your data back! That is a big reason I’m so hesitant to let go of ZFS for NAS use. I have abused the crap out of it and still was able to import arrays no matter what OS or hardware I used.

Especially automating backups…

I learned this the hard way about two years ago… I used a batch script in windows to backup files I didn’t have any permissions to on that user. So it created folders but no files in the folders… it was pretty much wasting power and giving me false peace of mind while it did absolutely nothing.

This is what got me.
I created a ZFS dataset to hold an rsync copy of the nas data. I never actually cronned the stupid job. Oh well. Live and learn, I guess.