Try ctrl+pageup to scroll up a page? The exact keystroke can vary depending on the terminal emulator, but something like that should do the trick.
that doesn’t work.
booting up my live session and running btrfs-check revealed that my newly reformatted partition is not a working BTRFS filesystem. it seems mkfs.btrfs does not yield a working btrfs filesystem. im unsure how to continue.
What is the kernel version on the recovery system VS the one on the server itself?
I’m beginning to suspect hardware
4.19.98 on both.
i was able to get a coherent BTRFS partition by reformatting to ext4 and then BTRFS.
my recovered files being written back onto my SSD is i type this.
Alright, let me know how it goes, I’m curious if it fails again. I would manually unmount the btrfs partition before shutting down to move the ssd back.
UPDATE: It Boots!!
but many things fail to start with invalid ELF header errors.
i believe this to be a separate issue and not related to my BTRFS corruption. this thread can be closed.
That’d be file corruption.
Try reinstalling the related packages, but that’s honestly not a good sign.
1 Like
i imaged the SSD again, then i decided to try the nuclear option. i wrote an apt script to redownload and reinstall all installed packages but not touch any configuration files, basically the inverse of a “factory reset”. a bit of slight troubleshooting and several hours of apt running, and i seem to have a fully-functional system again! just like before the power failure that caused all this.
UPDATE: it seems the VM image that is my webserver is also entirely corrupt. and so is its backup.
BTRFS couldn’t read tree root happens when it tries to boot.
btrfs restore doesnt work, same error. so i cant use the same strategy i used to fix my host.
does photorec work on the image at all?
photorec is unavailable in Debian Buster.
it looks like you just install testdisk and it should give you photorec as well?
https://packages.debian.org/buster/testdisk
I’m not sure why they did it that way but try anyway.
1 Like
photorec finds some files. nowhere near enough to rebuild my webserver VM.
You’ll likely have to rebuild the VM, but maybe you dont have to remake the content?
I assume anything that was in use at the time got mangled, which is rough.
it seems that the superblock is corrupted.
mount: /media/tmp: wrong fs type, bad option, bad superblock on /dev/loop2p2, missing codepage or helper program, or other error.
but BTRFS rescue super-recover seems to not be aware of it:
All Devices:
Device: id = 1, name = /dev/loop2p2
Before Recovering:
[All good supers]:
device name = /dev/loop2p2
superblock bytenr = 65536
device name = /dev/loop2p2
superblock bytenr = 67108864
[All bad supers]:
All supers are valid, no need to recover
attempting btrfs restore gives this error:
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
bad tree block 820707328, bytenr mismatch, want=820707328, have=0
Couldn't read tree root
Could not open root, trying backup super
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
checksum verify failed on 820707328 found E4E3BDB6 wanted 00000000
bad tree block 820707328, bytenr mismatch, want=820707328, have=0
Couldn't read tree root
Could not open root, trying backup super
ERROR: superblock bytenr 274877906944 is larger than device size 81068556288
Could not open root, trying backup super
so it seems the superblock is actually bad.
incase it helps, here is the output of btrfs-find-root:
btrfs root data
Couldn't read tree root
Superblock thinks the generation is 561502
Superblock thinks the level is 1
Well block 169705472(gen: 560517 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 167460864(gen: 560516 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 164200448(gen: 560515 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 149422080(gen: 560512 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 142884864(gen: 560510 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 138592256(gen: 560509 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 130531328(gen: 560507 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 127172608(gen: 560506 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 119816192(gen: 560504 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 108707840(gen: 560501 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 104644608(gen: 560500 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 97697792(gen: 560498 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 90292224(gen: 560496 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 86065152(gen: 560495 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 81920000(gen: 560494 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 78512128(gen: 560493 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 72925184(gen: 560492 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 67928064(gen: 560491 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 63406080(gen: 560490 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 60440576(gen: 560489 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 56606720(gen: 560488 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 51642368(gen: 560487 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 49102848(gen: 560486 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 44777472(gen: 560485 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 40091648(gen: 560484 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 37371904(gen: 560483 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
Well block 33046528(gen: 560482 level: 1) seems good, but generation/level doesn't match, want gen: 561502 level: 1
photorec was unable to recover anything usefull.
its almost all corrupted text files it found.
man, thats rough.
BIG F in the thread for your loss.
1 Like
im not convinced my VM is unrecoverable.
i believe im just not familiar enough with BTRFS.
I know Wendell’s been through this before with ZFS, but I’m not sure about BTRFS…
I’ll make sure he sees this thread.
2 Likes