bedHedd’s descent into fedora upgrade hell

This started off as a update post, but as I dove deeper I decided it was worth justifying a full post

Boy this upgrade process was a massive headache taking much longer than I expected. It didn’t help that I needed my laptop to be up and working the next day for a web conference.

update on the situation

Unfortunately in the middle of the upgrade, the insufficient space issues weren’t resolved. As a result, the upgrade was stuck and couldn’t finish, so I had to restart the computer. However restarting my laptop would interrupt the upgrade process and ultimately corrupt my package manager. I think a conf file that both DNF and YUM depended on was corrupted in the process of restarting, as when I ran dnf, it kept giving me a python libdnf import error.
With additional other errors

errors

: /

fedora libstdc++.so.6: version glibcxx_3.4.29 not found
lib64/libselinux.so.1 no version information availible required by sudo
importerror /usr/lib64/python3.8/site-packages/libdnf/conf.so: undefiend symbol

research

I tried a bunch of methods to fixing it

first solution: resolved python version issues

This was the closest, with a similar scenario. I didn’t want to risk removing a dependency. The easiest solution recommended was for me to back up files and reinstall Fedora.

If I had more time I would have tried the solution suggested here

second solution:use yum-deprecated

This was another solution, but I couldn’t reinstall yum-deprecated

third solution: reinstall dnf and packages

This was another source that had rpm commands, explanations for debugging issues

Looking back this answer would teach the user the commands to use to do dnf debugging. The only issue was I didn’t know how to download and install koji. You’ll see later I learn how to download stuff with terminal commands, but at this time I didn’t know how. This also plays into the second point in lessons learned.

fourth solution: download dnf/libselinux and pray to Linus Torvalds that uninstalling and reinstalling the two packages resolves library errors

With additional research, I decided to try running a curl/wget command on the libselinux to download and try uninstalling and reinstalling the package to resolve the libselinux error. I could figure out why I couldn’t curl/wget the package, turns out I had entered the incorrect url.
https://pkgs.org/download/libselinux
I was successful in downloading the packages, however when I went to install them, rpm refused to install it as the packages were already installed. At this point I should have gone back to solution 3 and try downloading and installing koji, but It was already 4 AM and I really wanted to sleep.

fifth solution replace the broken files

If the files are corrupted, why not replace them? The only issue I had was not knowing how to mount my flash drive from the terminal.
These guides were pretty helpful

I tried replacing the files, but still got the same issues…

sixth solution: backup and reinstall fedora

At this point it was 4AM, I was at my wits end, and I had exhausted my options and research. I was avoiding this from the start because I deemed it to be the brute force stupid solution and would have preferred to get more comfortable with the terminal,

but as they say

stupid is as stupid does

https://youtu.be/tldGgGFe194
And
It ain’t stupid if it works

Ultimately I could have revisited solution 2, but I really wanted a computer for web conferencing and wanted to sleep.

I ended up only getting one of those desires, so I will let you figure out which I got :upside_down_face:

sixth solution: you’d think it’d be easier

[it wasn’t]

backup

I first had to copy all my documents, downloads, videos, and pictures. Only thing was I didn’t know how to copy whole directories
I found this guide to be particularly helpful

reinstall

This took longer than I expected. I didn’t feel like setting up a new Fedora installation, so I reused a flash drive that had Fedora 28 (29 and 30 used a version of nouveau that would cause boot errors. I assumed I could jump from 28 to 32 in one update.

boy was I wrong

As learned later this would not be the case. Through trial and error, I learned that I could upgrade from fedora 28 to fedora 30, but not from fedora 30 to fedora 32. I had to wait through 2 upgrades before I could get to fedora 32. Once I got fedora 32 installed I went to sleep.

lessons learned

  1. I should have bit the bullet and install zoom temporarily on my phone/iPad to give myself more time, rest, and comeback with fresh mind instead of staying up all night.
  2. Learn how to setup a local a ssh session in the future from my laptop to my desktop, this would have made it easier to research, debug, and try out solutions. I researched was using my phone until I needed to charge it, and then continue on my iPad.
  3. I should have used my backup ssd to copy my Fedora files, and then experiment
  4. I learned how to mount drives from the shell
  5. I got more practice navigating, find commands, and file transfers in the shell.
5 Likes

if you dont skip any versions… you can go from 28 to 29 to 30 to 31 to 32 to 33

Ive done it

half the time with dnf… half the time with gnome software.

see the commands here to install the dnf upgrade plugin and just change the releasever to the next number.

2 Likes

The software center can be buggy sometimes. Even after clearing space out on the drive so it wouldn’t upgrade. DNF upgraded, but then it got stuck in the upgrade process :man_shrugging:

If you are more than 1 upgrade out from the latest… Then use dnf cli

1 Like

You could always just install Fedora on a spare flash drive and rsync the new /system partition to your target drive.

That way you don’t touch the /home partition, selinux and symlinks are preserved and you avoid all that nastiness above while also getting a “fresh” install with your data preserved

I employ this same method to use Fedora on filesystems that the installer doesn’t support (F2FS, ZFS etc.)

2 Likes

Yep that’s what I did : )

Huh I guess I’ll need to keep that in mind next time. Either way my documents and other files are backed up before I reinstalled.
If this method could preserve my customizations (ie all installed packages and anaconda environments), then it might actually be better than the brute force method I did in the OP

Can confirm it retains all your DE customizations. I never lose my gnome addons and themes across distribution upgrades, since those are all stored on the /home partition.

1 Like

how do I do that? Is there any guides to do that?

I can post my personal notes I wrote down for myself back when I had to scour the old fedora forums before they were purged.

Otherwise the current fedora documentation is extensive.

To put it simply: just install fedora to a spare flash drive
(we’ll call this slave drive for its purposes) with the regular fedora installer using the manual partitioning option. Then you want to manually partition your target drive you want Fedora on, exactly like you partitioned your slave drive using the command line or gparted. Either works fine.

Then you want to create mount points (directories) on the live session for the /root on both the target drive and slave drive, then mount them at each mount point.

From here you can rsync between both mounted partitions. Obviously you want to rsync the slave drive root partition you mounted earlier to the target drive root. Rinse and repeat for the /boot partition (important to copy over when upgrading whole distro), and /home if you need it.

Unmount each partition when finished, then lastly you want to chroot in to your target drive and change the /etc/fstab and /etc/crypttab (if you used encryption) as well as update grub with the UUID of your EFI partition on your target drive if you copied over the new /boot partition from the slave drive, at /boot/efi/EFI/Fedora/grub.cfg (per fedora documentation).

When finished you want to rebuild the grub.cfg using a command I can’t remember off the top of my head right now before exiting chroot (if you exit chroot by accident its no big deal you can just chroot back in without having to remount everything)

chroot doesn’t unmount partitions on fedora properly at this point so just systemctl reboot and let the system unmount everything.

At that point your system should boot up if you set the proper fstab, crypttab and grub.cfg uuid’s and and rebuilt the grub.cfg at /boot/efi/EFI/Fedora/grub.cfg while chrooted in the target drive.

For your purposes you may already have the partition on your target drive so you can just use gparted to wipe it or mkfs.x [mount point] (x is your filesystem) to wipe the partition on your target drive before you rsync the slave drive contents.

1 Like

Just an update;

The command to rebuild grub2 with a UEFI system:

sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg

Make sure while chrooted that you run “mount /boot/efi” before rebuilding grub so it’ll store the right stuff there

I can also walk you through on getting grub2 to work with secure boot if you’d like (keep in mind it doesn’t work with binary nvidia drivers)

1 Like

I’m going to read this again later.
I just bought a new m.2 drive (500gb) that I plan on switching with my m.2 (256 gb) drive on my desktop (only has 1 m.2 slot) and using the 120 gb drive that already has fedora installed as a intermediary test drive.

The plan is to replace the 256gb with my 500gb drive.

  1. Using my desktop I will first copy my 256gb drive to the 120gb drive.
  2. I will copy the 120gb to the 500gb drive using my laptop.
  3. I remove the 120gb drive and test the 500 gb to see if everything works
  4. I will swap out the 256gb drive on my desktop with the newly setup 500gb drive
  5. I will put the 256 gb drive in my laptop and use the 120gb drive as extra storage

This is what the red hat devs call the future?

Ha.

Glad I never used fedora.

1 Like

From personal experience with Fedora always keep a spare SSD around may it be for testing the new version or backup.

From reading all the issues you’ve had upgrading from Fedora is nearly the same hell I’ve experienced in the past but I’ve never waited that long between versions. Much of my issues is some stuff on Fedora’s repo sometimes disappears from one version as something didn’t meet the time frame between beta/RC. At one point I gave up using the Fedora Scientific Spin.

1 Like

it seems experiences vary… I have nearly a dozen fedora (server and workstation) machines & VMs that I run at home

and I’ve never had an upgrade issue

2 Likes

Turned this into step by step directions, will probably be asking questions based on the steps

  1. install fedora to a spare flash drive
    • (we’ll call this slave drive for its purposes) with the regular fedora installer using the manual partitioning option.
  2. Manually partition your target drive you want Fedora on
    • exactly like you partitioned your slave drive using the command line or gparted. Either works fine.
  3. Create mount points (directories) on the live session for the /root on both the target drive and slave drive, then mount them at each mount point.
  4. Rsync between both mounted partitions.
    • Obviously you want to rsync the slave drive root partition you mounted earlier to the target drive root.
  5. Rinse and repeat for the /boot partition (important to copy over when upgrading whole distro), and /home if you need it.
  6. Unmount each partition when finished, then lastly you want to chroot in to your target drive
  7. Change the /etc/fstab and /etc/crypttab (if you used encryption)
  8. Update grub with the UUID of your EFI partition on your target drive if you copied over the new /boot partition from the slave drive, at /boot/efi/EFI/Fedora/grub.cfg (per fedora documentation).
  9. chroot
  10. mount /boot/efi
  11. Rebuild the grub.cfg using a sudo grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg
    • chroot doesn’t unmount partitions on fedora properly at this point so just systemctl reboot and let the system unmount everything.

At that point your system should boot up if you set the proper fstab, crypttab and grub.cfg uuid’s and and rebuilt the grub.cfg at /boot/efi/EFI/Fedora/grub.cfg while chrooted in the target drive.

For your purposes you may already have the partition on your target drive so you can just use gparted to wipe it or mkfs.x [mount point] (x is your filesystem) to wipe the partition on your target drive before you rsync the slave drive contents.

+1, thank you, I might have to do this soon.

Now, with *buntus and btrfs, I rename @ and @home to something sensible, typically @r and @h, and adjust the fstab and grub accordingly. This frees up @ and @home for another install to the same btrfs, and sometimes I have half a dozen bootable installs happily coexisting in the same smallish SSD. Can I do this with Fedora? Or do I have to follow the above procedure?

1 Like