Messed up glibc on proxmox host, trying to save data from a VM

A couple of days ago I messed up my proxmox host by accidently installing an older version of glibc, I tried to fix it by linking libc.so.6 to a new version but no luck. On boot I get a kernel panic with this message.

/sbin/init: relocation error: /lib/x86_64-linux-gnu/libm.so.6: symbol __get_cpu_features, version GLIBC_PRIVATE not defined in file libc.so.6 with link time reference

Now I’m trying to save data from a VM that was not backed up, it is not the end of the world if I can’t rescue it but it would save a lot of time.

The VM in question was for testing and part of that was running cryptsetup on it so I can’t just mount it and grab the files from there. I can run qemu-system-x86_64 to boot the VM, get passed cryptsetup but I can’t get the data of it since it doesn’t have network access. I tried running this command

qemu-system-x86_64 -drive format=raw,file=/dev/pve/vm-202-disk-0 -m 4096 -net nic -net user,hostfwd=tcp::2222-:22

to get the data off it by running SSH/SCP on host to localhost -p 2222 but I get this error

ssh_exchange_identification: read: Connection reset by peer

I’m very much a beginner at linux so any help would be appreciated on how to get the data from the VM or if possible to get proxmox working again.

LUKS ? Do you have the headers backed up? If so, you can mount the drive elsewhere.

have you tried running

cryptsetup luksOpen /path/to/encrypted_drive my_encrypted_drive

then you have a drive at /dev/mapper/my_encrypted_drive

so you can then mount it !

mount /dev/mapper/my_encrypted_drive /path/to/mountpoint

should be good to go so nothing is lost !

LUKS ? Do you have the headers backed up? If so, you can mount the drive elsewhere.

Yeah, I meant LUKS. I don’t have a copy of the header but it should be still on the drive since I can unlock it.

cryptsetup luksOpen /path/to/encrypted_drive my_encrypted_drive

Yes, I have tried running luksOpen on the device at /dev/pve/vm-202-disk-0 but it doesn’t give any output or mount. When turning on --debug it says “LUKS2 header read failed (-22)”. I guess because it is trying to read from the boot part of the image? Here is a fdisk -l of the device:

Disk /dev/pve/vm-202-disk-0: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 524288 bytes / 524288 bytes
Disklabel type: dos
Disk identifier: 0xe62a7936

Device                   Boot      Start          End     Sectors      Size     d   Type
/dev/pve/vm-202-disk-0p1   *        2048         1499135    1497088     731M    83   Linux
/dev/pve/vm-202-disk-0p2            1501182    134215679   132714498   63.3G     5   Linux
/dev/pve/vm-202-disk-0p3            1501184    134215679   132714496   63.3G     5   Linux

Maybe if I could only target the /dev/pve/vm-202-disk0p2 partition?

Edit:

I mounted that partition by doing sudo losetup -o 768605184 /dev/loop10 /dev/pve/vm-202-disk-0 and tried luksOpen but I still get a “LUKS2 header read failed (-22)” error.

1 Like

It seems like I mounted the wrong partition in the previous message, I got it working by mounting all partitions with this command:
sudo losetup -Pf /dev/pve/vm-202-disk-0
I tried sudo cryptsetup isLuks <partition> on the new devices in /dev/ and found the partition.

The partition contained an LVM2 member so I scanned for new LVM devices (sudo pvscan and sudo vgscan) and mounted the new active LVM volume.

I can now access the VM data :slight_smile:

Lesson learned: Keep backups of everything, even the testing VMs

2 Likes

Sorry I wasn’t on earlier… glad youi have the data and solution.

side note: from the command I posted, the LUKS drive should have been mounted at /dev/mapper/my_encrypted_drive which you then would mount it to a directory of your choice. I felt i wasn’t that clear in my instructions but assumed you would understand. the commands get tricky to read out and thats my fault for not being more descriptive.

All in all glad it worked !