Totally broke Truenas scale 23.10 by upgrading bootpool, with no backup

Hi. First of all, i’m pretty new to linux/truenas/zfs, and I fucked up.
So I recently upgraded to truenas scale 23.10 (and 23.10.1). I ran it fine for a few days, and then decided to upgrade my main pool by running:
zpool upgrade -a
Even after that everything was fine, until I needed to reboot… Now I knew then that this would also upgrade my boot-pool, but what I didn’t relize is that wouldn’t allow me to boot past the grub cli. I’m now stuck at some sort of grup cli, with the error “file system not recognised”.

So for example “ls (hd0,1)/” just returns “unkown file system”.

Is there someway I can bootleg truenas to launch even once so I could backup my configuration, and reinstall truenas scale? I kinda figure that the bootloader doesn’t support openzfs 2.2, and I messed up, but If there is anything I can to retain my old config, any help would be greatly appreciated.
And if this is the wrong place to post this, I’m sorry. If that is the case, could someone point me in the right direction to find some help.

Oh man, I had this same issue and posted about it in the TrueNAS Community forums: 23.10.0 GRUB boot screen after ZFS upgrade | TrueNAS Community

You can upgrade any zpool in TrueNAS SCALE except the boot-pool.

When loading the image onto a USB drive, you also can’t fix it that way.

The solution I had was to take my backup file and import it on to a completely fresh install.

Since you have no backup, your best bet is to grab the config out of your boot-pool. I was able to mount my boot-pool with GRUB somehow, but I didn’t know how to pull the data off.

Using another Linux distro with ZFS might be your best bet, like a live environment of Proxmox. You should be able to do it from there.

Good job, iX Systems.

Proxmox has no live environment. I’d get a Core ISO and manually dig out the config file from the boot disk. Question is if Core can import the updated boot pool…Proxmox ZFS version is usually ahead of iX Systems, so Proxmox should be a safe bet, but you have to install it.

My 7.4 Proxmox runs on “zfs-2.1.11-pve1”. Not sure what zfs version Proxmox 8 is running but I guess it’s higher than that. Yeah, I should upgrade…but why upgrade if nothing is broken? I could end up in GRUB rescue mode :wink:

1 Like

Than you so much for your answer, I wasn’t able to find that thread on my own. Yeah, I did read about that, sadly after I had already upgraded my boot pool. Also the only file I was able to find in grub was some efi stuff. I’d think that ubuntu 23.10 might work for that . It’s at least the easiest thing to try for me right now (the info I found quickly points to something like openzfs 2.2.0rc3 or something wierd like that)

Or just re-install Scale. Saves a lot of trouble and within 15min you can redo all shares and configs.

I could do my configs faster than installing Ubuntu, getting the files and stuff…

Or get something different than Scale. Bugs are more on the rise than declining. I won’t touch Scale until 2025. Proxmox and Core ftw.

I mean I have a pretty complex apps setup, which I kind of wish I had done In proxmox now, so it would take a bit more time. And I think I can create a live ubuntu that should be able to mount my boot-pool quicker. I’d also have to kind of “reverse engineer” a few iscsi shares. But thanks for the suggestion

Check ZFS version. 2.2.0 has some bugs around block cloning. I’d stay away from 2.2.0 for at least a month or two unless you really need some features. Leading-edge on your filesystem can be dangerous, not only for Arch users