Picked up some drives at micro center when it opened. Headed in to install them now. Destination NAS is at 96%. Netcat is still running so should be ok. My main concern was if it failed and I had to start over.
Well, drives are in now, but netcat appears to be frozen… the processes are going on each end, but there’s no data being sent.
why you using netcat?
@SgtAwesomesauce recommended it because I was worried about network bottleneck. It appears to be working well and on an acceptable timeline so I’m sticking with it.
It froze earlier because the destination array had filled up. I didn’t make it in time with the new drives. In any case, it should finish in the morning.
Damn dude, this plays out like a movie.
Did it wind up needing to be restarted, or did the transfer just pause? I’ve never encountered your issue before, so I’m very curious.
I deleted the image and restarted it. I think ZFS just had a pending write that was never going to go through, but that’s a guess.
It’s done now. I’m heading in to boot into Ubuntu live, mount the ZFS volume where the image is and try to recover with ext4magic and/or extundelete.
Well, the copy was successful. I snapshotted it, and now I’m in Ubuntu live and can mount the image file.
Unfortunately, ext4magic returns:
Error 2133571465 while reading block bitmap
So I ran e2fsck on it and get a ton of messages:
INDEX_FL flag set on filesystem without htree support
extent tree could be narrower/shorter
free blocks count wrong
and others…
ext4magic returns the same error after e2fsck -y (remember I have the initial copy snapshotted)
Hmm, that’s really odd. Looks like its corrupt. you were dding from an offline volume, right?
Yes, and it mounts fine…
Corruption could have been the culprit all along. Maybe time to go to testdisk…
Well EXT4 is a very interesting filesystem.
It allows you to mount if a certain small subset of data structures still exist, but will throw tons of errors along the way and you may or may not have all your data.
Does the QNAP do SMART monitoring? I’m starting to wonder if a disk went bad.
Additionally, you might find more success with Foremost. Do you have the space to testdisk that array?
Yeah, QNAP does SMART. No errors.
What are the space requirements for testdisk? Do I need to be able to copy the whole thing?
I tried extundelete but it apparently doesn’t play nice with e2fsprogs after version 1.42. So time to switch to Ubuntu 16.04… maybe that’s the issue with ext4magic as well?
Potentially.
This is why I have a stupidly large array.
Possibly… That’s interesting. Did they say why it doesn’t play nice?
https://bugs.archlinux.org/task/56585
I’m trying Ubuntu 16.04 now… actually installing it on a flash drive. I’m tired of dealing with the live usb.
I have a 500GB USB M.2 enclosure that’s got a few different distros installed on it. Really helps for when I run into similar issues.
Yeah, I need to set something like that up. Right now I have a keychain with about 20 flash drives on it that are labelled. They’re mostly install media though and not bootable OS’s.
Do you just have a bunch of boot entries in GRUB or do you manage it another way?
Grub will chainload an ISO. Super convenient
Yeah, it’s a bunch of ISOs and a couple installed environments (Kali, Lubuntu 14.04)
$110 for a solid solution.
Have you tried NetBoot.xyz? That covers 90% of my needs.
I haven’t even heard of it.
Most of my needs are in various datacenters when we have various levels of connectivity, so I like to have all the data locally.
Pro tip: some Supermicro Servers won’t boot properly from USB device if more than 3 usb drives are plugged in. It dumps a bunch of ACPI errors and you get an initramfs shell. Presumably the usb bus can’t fully power all of them at once.
Yeah, I’ve seen that.
It also doesn’t like my SSD array, possibly because it’s multipathed through a backplane to separate HBAs.