Between checking the mint forums and my question on ubuntu stackexchange, I saw 3 people with the same hardware (sandy bridge) having the same/similar problems.
Not sure if buggy UEFI or what, 3 independent instances seems more than a coincidence at this point.
Also, I don’t think my bios can even switch between UEFI. Scouring the menus, I didn’t find any setting to that effect.
Honestly this is sounding more and more like the kind of insane heisenbugs you only hear about in other people’s stories.
You cannot switch from uefi to bios, you need to find so called Compatibility Support Module. This setting enables legacy bios emulation.
There should be switch somewhere, but if you installed your original os without CSM enabled, you wont be able to boot afterwards anyway. It seem like red herring.
Issues with secure boot being on and enforcing would have caused issues much earlier, so its also red herring.
Your problem is stated clearly initramfs unpacking failed invalid magic at start of compressed archive → either your initramfs is corrupt or it is compressed by unexpected and unloaded method at this point.
Either check your environment setting and regenerate your initramfs in liveusb via chroot or bypass initramfs entirely and boot your kernel directly by editing kexec options at runtime.
This is easy if you dont use encryption/raid or exotic fs. Unfortunately I think btrfs falls into the last category.
So best way to do this would be liveusb, chroot to old install and then trying to suss out what went wrong and backing up personal data in case of reinstall being needed.
I was more thinking if the bios was in legacy, maybe the upgrade made a buggy assumption and tried creating the initramfs assuming it was uefi or… something. “Invalid magic at start” makes me think along those lines anyway.
a) The kernel is way above the required version for zstd. (Ubuntu and Mint come with 6.8. I was on 6.8 before that anyway.)
b) I don’t have /etc/mkinitcpio.conf (Maybe ubuntu has a different boot process that doesn’t use mkinitcpio?)
(I think I saw this forum post already, but the above mean it doesn’t apply in this case.)
Maybe it also has something to do with how the installer sets up btrfs subvolumes? ¯_(ツ)_/¯
I still have a backup image of the borked system, but I’m already almost done setting up a new Mint 21 install, so at this point it’s academic curiosity.
Same issue, different toolset on difference distro. Find relevant config and do whats needed to change used compression algorithm back to whatever your kernel can actually handle right now.
For example the kernel is still a version that understands zstd.
Not to mention, I don’t understand what is being compressed here?
@greatnull You seem to know this stuff. Are you able to explain the process?
As far as I know the boot process goes like this:
Power on
The BIOS keeps the boot order in CMOS memory (aka BIOS NVRAM). From there it chooses a boot device (e.g. your hard drive).
It looks for that drive’s boot sector. UEFI understands GPT (a type of partition table), so it can boot off GPT formatted drives. *
The boot sector tells it how to load the boot loader (altho in UEFI, LBA 0 is actually a protective MBR, but anyway…)
BIOS loads the boot loader - GRUB, for linux - and hands off control to it.
GRUB then loads the kernel.
Kernel brings the system to usable state…
Step 5 and above I have less knowledge of.
Been googling and looking at wikipedia, but the order of events and function of various steps are still not clear to me. This page gives the rationale.
So where exactly does the error occur? During grub? Or during kernel init?
Kernel 6.8 that ubuntu 24 and mint 22 load understands zstd.
Heck, Mint 21 comes with Kernel 5.15 and can boot with zstd.
Altho Mint 21/Ubuntu 22 has GRUB 2.06 and Ubuntu 24 → GRUB 2.12.
At the same time, restoring the pre-upgrade snapshot does not fix this error, which I fail to grasp how it can be possible…
So help me understand where in the whole process the error arises.
Also, what’s the least disruptive way to test boot-related stuff like this? I already reinstalled into Mint 21 and have it mostly set up. But I’m still super curious about getting to the bottom of this error. Not just fixing it, but understanding why/how it happens in the first place.
I only have gig networking so tossing around drive images that weigh 512 gigs is painful. Do I just save home and /etc? Maybe /var too?
* There is a cool tool called DMDE that can be used to read and parse a disk sector by sector and troubleshoot partitions and volumes.
…
6 . Grub loads the kernel from /boot partition 7. Kernel extracts initrd as initial ram filesystem (intermediate root enviroment) and mounts it 8. Kernel and tools from intermidiate boot envirement initialize system and try to mount and switch to real root
9. Init system is started
10. System start everything needed to reach target runlevel → usable interactive state
… Note: its possible that order of 8 and 9 is wrong.
You are definitely encountering issues at steps 7 and 8. I havent mucked with this for a long time.
Step 7 fails, because kernel is not capable of decompressing initramfs, hence hard error. This can easily happen as mentioned in issue I linked.
If you try to bypass initramfs (its not needed for all setups), you fail at step 8, since your kernel does not seem to have btrfs support enabled at this boot stage. I think that the module must be compiled in. You can see list of supported FS in the screenshot.
Necessary modules for mounting btrfs are pretty likely in initramfs, hence your failure to boot.
So your fix is really easy, boot into liveusb, chroot to your install, look up where are setting to your initrd creation process is defined, change compression to something basic or disable it and then regenerate initramfs.
/boot/initrd.img is a compressed image of a file system that is loaded into RAM. This is part of an intermediate step in the booting process, the rationale for which is explained here.
The failure comes from the kernel failing to extract this compressed image into RAM.
But why?
Ubuntu 22’s kernel can do it.
Did 24 compile the kernel without zstd support? And how does my hardware play a role? Why aren’t more people encountering this error? Why does restoring the pre-update snapshot not fix the system (the snapshot does contain the boot directory, along with the old kernels and old ramdisk images)?
You also said
But the kernel loads initrd.img, not the bootloader, right? At that point control has already been handed to the kernel, right?
Also, I still have no easy way to play around with this and then restoring back to my already set-up system…
Error on my part, bootloader has modular decompression support if kernel image is compressed, and thats not relevant here.
But there has been kick over last few yeas to standardize modern lz4 or zstd as default compression for kernel and initramfs.
But why?
Ubuntu 22’s kernel can do it.
Did 24 compile the kernel without zstd support? And how does my hardware play a role? Why aren’t more people encountering this error? Why does restoring the pre-update snapshot not fix the system (the snapshot does contain the boot directory, along with the old kernels and old ramdisk images)?
No idea, ask ubuntu folks, I am firmly in red hat territory since forever. Did you performs update following the best practices to the letter? Did you customize your 22 install in any significant way, especially kernel, initrd or related config?
Its important on less complex debian installs, so I suppose same applies here.
Haven had failed system update in 15 years with fedora.
As to your hardware, it should not play any role whatsoever. There certainly inst anything pointing that way in my opinion.
Yes, and no. More importantly, a squeaky-clean (dd-wipe efi partition) fresh install of Mint 22 (ubuntu 24 base) also failed to boot in the same manner.
I only bring up hardware because 2 other people in 2 different places, with the same hardware seem to be experiencing this or a similar error.
Also, still taking ideas on how to more easily back up my current system state and play around with this error.
Edit: Actually df tells me /dev/sda1 is mounted on /boot/efi (not /boot, which I had thought until now). And /boot/efi is not part of the pre-update snapshot. So by method of exclusion the error must originate somewhere in the bootloader part.
Seems more likely that ubuntu 24 base is more likely culprit here. I cant imagine hardware issue capable of causing this that would be blindingly obvious everywhere else.
Either initramfs was corrupted during update process, or compression/decompression routines are somehow generating gibberish, now or then. Both would point to systemic issue.
I would ignore the hardware until normal recovery and fully clean reinstall fails outright.
Liveusb + external drive is easiest way to do both backup and try fixing the original install.
