I created a ZFS pool (with great originality, named tank) and started copying over files from my old degraded NAS with the intent of importing this pool into a potential TrueNAS server in the future (pending a system upgrade on my end) using mv (I know, bad idea, I should’ve used rsync -a instead) and we had a brief electrical outage.
Rebooting back into my system and tank no longer exists, the mount point /tank does but now it’s just a folder.
$ zdb -l /dev/disk/by-id/ata-STXXXXXXXXXX-XXXXXX_XXXXXXX
failed to unpack label 0
failed to unpack label 1
failed to unpack label 2
failed to unpack label 3
$ zpool import -a -f -d /dev/disk/by-id/ata-STXXXXXXXXXX-XXXXXX_XXXXXXX
no pools available to import
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
[...]
sdc 8:32 0 3.6T 0 disk
├─sdc1 8:33 0 3.6T 0 part
└─sdc9 8:41 0 8M 0 part
[...]
$ zpool import -f tank
cannot import 'tank': no such pool available
$ zpool import -N tank
cannot import 'tank': no such pool available
$ zpool import tank /dev/sdc
cannot import 'tank': no such pool available
$ zpool import tank /dev/disk/by-id/ata-STXXXXXXXXXX-XXXXXX_XXXXXXX
cannot import 'tank': no such pool available
did you maybe point ZFS at the whole devices while leaving the partition table backup at the end of device ontouched, and they eventually got restored on top of ZFS superblock somehow … really weird stuff
This create a single disk zfs pool, if so where did /dev/sdc come from?
Ignore huge numbers like this, this is seagate storing multi-byte values.
I’ve had this indicate a bad sata cable in the past, but if it’s only1 then it’s probably pure chance.
I don’t see anything obviously bad in the smartctl output, so let’s assume that at least that disk is fine. How do the other disks look?
This blkid output is a bit weird, it’s showing linux_raid_member and zfs. Did you reuse a disk for zfs that was previous mdraid, or did you try doing mdraid on top of zfs?
Either way this implies that the partition/disk is part of mdraid and not zfs. Is /dev/sdc definitely the disk you used for zfs?
… never tried giving it raw devices due to bad experiences with bootloaders in the past . I always point at a partition and I try to leave enough buffer at the beginning and end of disk (I pinch 1G on either side).
Apparently this 8M reservation is some weird solaris thing, and apparently normal despite looking confusing to me; good to know.
So actual zfs is on /sdX1 I’m guessing, so zdb should let you walk the data structures on this auto-created partition.
/dev/sdc was /dev/sda at one point, it changed because I inserted an SSD into my hot swap bay and I fetched the command from my .bash_history file from a time before that.
The disk was formerly part of a RAID1 array on a WD My Cloud EX2 Ultra (hence the label “Vanilla” that for some reason never went away) , I formatted the entire drive and then created a new GPT partition table but “Vanilla” still stayed as a label so I figured it was just Debian derping out and that it wasn’t representative of what’s on disk.
Also, I presumed it was reporting it as linux_raid because perhaps GParted doesn’t know how to interpret ZFS? Maybe that was foreshadowing that something was gonna go wrong
In my defense, I did try to recreate the zpool thrice (after recreating a new partition table in GParted each time) and it still retained itself the label of linux_raid
It’s possible that the data is still there, but linux is only loading the first filesystem it finds. (Very simply there are magic bytes that identify filesystems, and it’s possible for the magic bytes for one filesystem to still exist after a quick reformat with a different filesystem).
Can you try running this command to read and report the filesystems that the system can find on the disk? (The command is called wipefs, but by default it doesn’t do any wiping - it’s very badly named).
I always figured that stuff is stored in the partition table à la partition type GUIDs and that any magic bytes would be ignored subsequent to a new parition table being created.
That being said…
$ wipefs -n /dev/sdc*
DEVICE OFFSET TYPE UUID LABEL
sdc 0x200 gpt
sdc 0x3a3817d5e00 gpt
sdc 0x1fe PMBR
sdc1 0x1000 linux_raid_member 424be1fb-d239-1bce-ba39-38c2ecea6031 Vanilla:0
sdc1 0x3f000 zfs_member 8199371577609406066 tank
[...]
sdc1 0x60000 zfs_member 8199371577609406066 tank
sdc1 0x3a380dbf000 zfs_member 8199371577609406066 tank
[...]
sdc1 0x3a380de0000 zfs_member 8199371577609406066 tank
…I’m feeling pretty stupid right now. Rewriting the whole disk with zeros is time consuming and I had limited time but more time has been spent trying to fix something because I was trying to save time.
I would try zpool import -Dzpool import With no drive address (nor letters) When it still comes up blank, I would just create a new pool, old one trashed. If it does not come up blank, might need -N, (non destructive check if it can import) and possibly -F / -X in case it needs to dump transactions. Check the manual for what each does first.
It’s just a single drive, so is only for transient/temporary data.
I’m not knocking a single drive pool; I have a laptop with a single drive pool, and a pi.
But they are doomed to die an unrecoverable death, and when they do, no tear shall be shed.
As for initial population, if the source is an existing zfs pool, one could zfs send/receive the required datasets/pool with all the snapshots required?
The patition1 and part9 is normal Solaris behaviour. As is keeping disk labels in tact. Leaving the raid member was a bit odd, but I’ve not used a raid member as part of a pool.
ZFS *Should write a bunch of super blocks through the drive (maybe just 3?) to allow for partial disk write over, but I don’t know if it requires another part of a vdev to validate what is good on a resilver (as in, needing to be part of at least a mirror)
When switching filesystems at the very least you can use wipefs to remove magic bytes (use the -a option to do the dangerous stuff), or use dd to zero the first 4MB or so - most magic bytes are stored there. Though some filesystems also have a footer, like some forms of mdraid.
Partition ids are basically ignored by windows and linux, I think some BSDs and Mac OS still look at them though. Most reformats will wipe the old headers, but mdraid is probably using a footer here.
Okay so I’m of two minds. On one hand the mdraid may have mounted automatically and tried to “correct” any problems wiping your data (check /proc/mdstat), on the other hand it may just mount if you remove that linux_raid_member header. If you really want to try this then make a backup of the whole disk, and read the man page of wipefs on how to remove a single signature. Removing that lvm_raid_member signature may do something. Like, I’d expect zfs to find its super blocks and import the pool, but it may be trusting the blkid identification on linux? At this point assume your data is missing.
The next step is to try running an “undelete” program on your old drive, the data you moved may still be there.
When switching filesystems at the very least you can use wipefs to remove magic bytes (use the -a option to do the dangerous stuff), or use dd to zero the first 4MB or so - most magic bytes are stored there. Though some filesystems also have a footer, like some forms of mdraid.
Will keep this mind in the future, didn’t know that non-standard behavior was to be expected, kinda hoped GParted would do this for me by default or would at least suggest me the option if it saw a linux_raid_member and knew of these shenanigans.
At this point assume your data is missing.
Presuming mv doesn’t delete anything until everything is copied, nothing is lost. Speaking of which, does it? If it does, I’ll just zero out the first and last gig of the drive, quick format and run rsync -a like I should’ve. If it doesn’t, I’ll need to recover that data.
I’ve run wipefs -fo 0x1000 /dev/sdc1 (the offset specified in the output of wipefs -n /dev/sdc*) and verified that the signature is gone using wipefs -t linux_raid_member -f /dev/sdc* (and by opening GParted, that now reports it as ZFS)
Still didn’t work. Tried all my above commands, a system reboot, a reinstallation of ZFS.
Fuck it, time to start clean.
Here’s a little command I wrote…
function scrubby {
SCRUB_QUANTITY=2048000 # 512*2048000 = 1.0 GB
[[ $EUID > 0 ]] && echo "$0: This tool must be run as superuser" && return 1;
[ -z "$1" ] && echo "$0: No block device specified (e.g. /dev/sdx)" && return 1;
dd if=/dev/zero of="$1" bs=512 count=$SCRUB_QUANTITY;
dd if=/dev/zero of="$1" bs=512 seek=$(( $(blockdev --getsz "$1") - $SCRUB_QUANTITY )) count=$SCRUB_QUANTITY;
return 0;
}
$ zpool create tank /dev/disk/by-id/ata-STXXXXXXXXXX-XXXXXX_XXXXXXX
/dev/disk/by-id/ata-STXXXXXXXXXX-XXXXXX_XXXXXXX is in use and contains a unknown filesystem.
The lsof is empty and so is my patience. I’ll just make a plain, boring partition and move on and let future projects be a subject of exactly that.
Thank you everyone for your help! I learned a thing or three about ZFS, even if I didn’t want to right now but I’m running out of time.
