VM migration between QEMU KVM versions [SOLVED]

[ro55mo]

Hi

So a have a couple of 8GB Raspberry Pi 4’s running Debian 10 acting as KVM hypervisors and all is well.

They have shared network storage and I can happily live migrate my VMs between them.

I have a third Pi of the same model which is now running Debian 11. KVM is installed and I can create and run VMs on the same shared storage.

I want to migrate a VM from the initial pair of Pis running KVM on Debian 10 to the the Pi running Debain 11.

If I do this as a live migration I get the following.

If I shut the VM down and delete it while retaining the disk and then set a new VM on the Debian 11 host using the same disk I get the following.

I might be able to fix the latter but I am stumped by the former.

This raises the question however, how does a person move a VM between QEMU KVM versions on different hosts? Either live or cold.

Surely their must be a way or how else can you upgrade or rebuild your hypervisor hosts?

It is possible on ESXi I know for sure.

Any information or background would be gratefully received.

ro55mo

[imhigh.today]

I get the same issue on Debian Buster when migrating from an Intel Xeon to AMD Epyc machines.

Loading a LiveCD and reinstalling Grub in the VM is the work around. I don’t know why EFI fails to find a perfectly valid disk once the underlying machine changes, but it’s not destructive to your data.

It’s also why I completely ignore the existence of EFI on virtual machines and never deviate from the default BIOS setup.

[modzilla]

Migration between qemu versions is normally fine, you’re just not able to migrate back to a lower version. Offline migration should always work tho.

@ro55mo

This might be because you did not copy over the nvram of the virtual machine that contains the information on what EFI the UEFI should load on boot.

If you enter the following commands, the vm should just boot:

fs0:
cd EFI
dir
cd $(The directory in question)
dir
$( sth.efi most sth like grubarm64.efi)

Normally the fallback for a EFI-System to boot into would be EFI\BOOT\BOOTX64.EFI, but since you’re on arm, I don’t know if that’s the same or it’s something like BOOTARM64.EFI.

To make to boot entry persistent you’ll need to create a new one with efibootmgr:

efibootmgr -c -d /dev/$(Presumebly vda or sda) -p $(Your boot Partition fdisk -l /dev/vda) -L BOOTMEWILLYA -l \EFI\$(The folder from before)\grubx64.efi

[modzilla]

Or you could just copy over the following file from the old to the new pi

I completely forgot that you could just do it in the BIOS as well XD

The video is made in HEVC so, you’ll need to open the videourl with VLC, hope you don’t mind

[ro55mo]

Copying the NVRAM file to the same location on the new Pi was the solution.

So, in summary.

1. Backup your data!

2. Shutdown your VM.

3. Clone your VM (as another fall back).

4. Copy the nvram file from /var/lib/libvirt/qemu/nvram/vm-in-question.fd on source hypervisor to target hypervisor.

5. Delete your original VM but not its disk file.

6. chmod the permissions of the nvram file to 600 for root on the new hypervisor.

7. Create a new VM using the same settings as the original, CPU, RAM, network plus the original disk file.

8. Power on.

9. Success.