Virtualizing Rpi on Ampere/Ubuntu 24.04 aarch64

Operating Systems & Open Source Virtualization
, , , ,
1

I have a System76 Astra (ALTRAD8UD2-1L2Q) with lots of space and ram. None of that is doing me much good if the Rpi5 system images I want to virtualize with either give me a VGA or serial console. Of course, I cannot just boot a plain Rpi5 image, that would be #tooeasy. When enabling the VGA driver I just get the TianoCore boot shell :frowning: It does not see the qcow2 filesystem as a boot device. If I ls fs0: I see some files, but the TianoCore BIOS screen never lists a bootable file system.

Am I missing a .efi driver somewhere?

For console:

  • video=none
  • Copied the initrd and the kernel from the qcow2 file
  • defined the extra dtd
  • updated cmdline.txt to add console=ttyAMA0,115200
  • updated config.txt to add enable_uart=1

With some combinations I get a serial connection but no login prompt, but mostly I don’t get any sign of boot.

2025-02-04_21-55
2025-02-04_21-551012×666 96.9 KB

Console 1:

<console type="pty" tty="/dev/pts/6">
  <source path="/dev/pts/6"/>
  <target type="virtio" port="0"/>
  <alias name="console0"/>
</console>

Console VirtIO Serial 0:

<controller type="virtio-serial" index="0">
  <alias name="virtio-serial0"/>
  <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>

And I’ve tried defining the dtb for the bcm2712-rpi-5-b:

  <os firmware="efi">
    <type arch="aarch64" machine="virt-8.2">hvm</type>
    <firmware>
      <feature enabled="yes" name="enrolled-keys"/>
      <feature enabled="yes" name="secure-boot"/>
    </firmware>
    <loader readonly="yes" type="pflash">/usr/share/AAVMF/AAVMF_CODE.ms.fd</loader>
    <nvram template="/usr/share/AAVMF/AAVMF_VARS.ms.fd">/var/lib/libvirt/qemu/nvram/2501-rpi5-build_VARS.fd</nvram>
    <kernel>/VMs/l_2501-rpi5-build/kernel_2712.img</kernel>
    <initrd>/VMs/l_2501-rpi5-build/initramfs_2712</initrd>
    <cmdline>rw earlyprintk loglevel=8 root=/dev/vda1 rootdelay=1 console=ttyAMA0,115200 rootfstype=ext4</cmdline>
    <dtb>/VMs/l_2501-rpi5-build/bcm2712-rpi-5-b.dtb</dtb>
    <boot dev="hd"/>
    <bootmenu enable="yes"/>
  </os>

Also, I’ve been referencing Perplexity: https://www.perplexity.ai/search/i-am-running-an-arm64-server-i-QG9Ir_3iQUWrXgl9uBjVzw
Search results on S/O mostly seem to focus on amd64 based emulation (ok)…and after looking at so much of this I’m not sure there’s much of any difference by being on a fancy ARM CPU. #gottabeabetterway

Virtualizing multiple raspian installs on Pi5
2

I’m pretty sure that kernel supplied with Raspberry Pi OS image is tailored for booting from device bootloader, not a tianocore uefi. If you have a real device - go check the kernel config it was built with. I wouldn’t be surprised if there are no VIRTIO or EFI related options selected. The machine type “virt-8.2” isn’t emulating any of the devices the RPi image expects.

1 Like
3

That seems sound. I also found the Ubuntu aarch64 QEMU guide. I will work through that first.

4

I wonder if the Ubuntu blog post author will respond:

5

OK this blog post seems to be useful:
https://blog.jitendrapatro.me/emulating-aarch64arm64-with-qemu-part-1/#The-Basics

6

emulating aarch64 using machine=virt is stunningly slow. Just installing packages from Debian 12 netboot not only takes like 3+ hours, but also fails. :frowning:

7

Making progress! I have been following the blog.jitendrapatro.me guide.
I was able to start netboot and get a qcow2 file installed with a system image.

A large part of the problem is that the virt-manager interface is does not entirely correspond to what the qemu-system command arguments look like. Using the qemu-system-aarch64 command to get this booted, I could then start converting the next command into libvirtd xml:

Install.sh

#!/bin/bash
qemu-system-aarch64 \
    -machine virt-8.2 \
    -cpu neoverse-n1 \
    -smp 4,sockets=1,cores=4,threads=1 \
    -m 3072m \
    -initrd initrd.gz \
    -kernel linux \
    -drive file=2504-debian-12-10-root.qcow2,if=virtio \
    -nic user,model=e1000 \
    -nographic

You have to hard-kill the qemu-system-aarch64 process after reboot. Then copy the kernel and initrd out. I could then make this run.sh:

qemu-system-aarch64 \
    -machine virt-8.2 \
    -cpu neoverse-n1 \
    -smp 4,sockets=1,cores=4,threads=1 \
    -m 3072m \
    -initrd initrd.img-6.1.0-30-arm64 \
    -kernel vmlinuz-6.1.0-30-arm64 \
    -device virtio-scsi-device \
    -device scsi-hd,drive=d0 \
    -device scsi-hd,drive=d1 \
    -device scsi-hd,drive=d2 \
    -drive id=d0,if=none,index=1,file=2504-debian-12-10-root.qcow2 \
    -drive id=d1,if=none,index=2,file=2504-debian-12-20-swap.qcow2 \
    -drive id=d2,if=none,index=3,file=2504-debian-12-30-home.qcow2 \
    -nic user,model=e1000 \
    -nographic

At that point I could ask perplexity.ai to translate that into XML and then compare what virt-manager was crafting for me. Notably:

  • disks need to be SCSI
  • remove ACPI
  • remove EFI/UEFI declarations
  • remove TPM

Another important distinction is that the TianoCore BIOS does not know anything about VirtIO. No wonder none of my disks were seen! the TianoCore will see some SCSI disks, but it cannot do anything with them :frowning: Possibly because they are not GPT formatted. But because this needs to do a direct-kernel boot, no GRUB or /boot is necessary.

8

…and there I go…numbly doing a fsck.ext4 fs over the wrong virtual disk :face_with_diagonal_mouth:

9

so use the UUIDs because I just learned that the /dev/sdX order isn’t stable (I should have known)

1 Like
10

Interesting project, I do kinda like to hear if you succeed. I would help but I do not have an ARM based system, apart from an actual RPI, here to test anything. If you do I would kindly start an attempt to follow your setup but with the difference of emulating aarch64 on x86, but since that will be an additional step I’d have to do I’d like to see if you manage to get somewhere.

11

I am making progress. I thought I would meet with a friendlier virt-manager experience but that it insisted on adding non-appropriate defaults really threw me. It was necessary for me to see the extra xml that was getting added, and rip it out.

12

Hi @memnoch_proxy. I’ve got some time to try this myself and it’s been a good few years since I’ve touched libvirt.

Testing on a 8gb rpi5 device. Made a remote headless libvird service listening to TCP, to drive it from the main PC.

Grabbed Ubuntu 24.04 cloud image and it worked out of box with minimal emulated hardware:

  • UEFI firmware for arm64 architecture (debian package qemu-efi-aarch64)
  • removed GPU, mouse, keyboard devices, spice
  • image is qcow2, thus used it as a backing store for expanded 10Gb virtio disk.

See the recording

Edit:
Tried same for Debian 12 nocloud image - also worked without XML editing. Here is the recording.

2 Likes
13

Regarding booting the vanilla Raspberry Pi OS image - unfortunately not possible. Just like I thought - image is tailored for booting on the real rpi hardware:

  1. The only compiled in PCIe controller is the Broadcom’s and no virtio drivers are present at all, not even as a loadable modules. Thus we need a custom kernel.
  2. Image itself is comprised of two partitions, labeled bootfs and rootfs, formatted under the legacy MBR table. That’s why the disk image is invisible to the UEFI firmware - those expect only GPT partition table. Thus we should boot under BIOS emulation, not UEFI.
  3. Image comes with initramfs which should handle the first boot and some initial system configuration, we can ignore that for now.
  4. Image has pre-defined user account named pi, but it’s default password is not set, thus even in case of successful boot - we wouldn’t be able to login. Thus we would have to first mangle with /etc/shadow file on the rootfs partition and clear the password.

After a bit of back and forth, I’ve managed to boot the almost pristine Raspberry Pi OS Lite 64-bit image with custom kernel and mostly default libvirt’s domain.xml. See the screencast

2 Likes
14

finally got time to watch those. Appreciate you taking the time to make them. Definitely there’s a lot of features to remove from a system to get those to boot. I’m inspired to give it another try.

15

ok–this custom kernel. I had been copying the Deb12 kernel and initrd over from my booting bookworm VM. This must not be a custom enough kernel? I’ve tried booting the bookworm netinstall arm64 kernel file and initrd, and also the the bookworm 6.1.0-30 kernel and initrd, neither of which seem to boot at all when paired with the rpios filesystem. I must be assuming too much of these, but I’m not clear what.

What puzzles me the most is that the Console device never outputs anything. I’ll post screenshots after I eat.

16

I’m not sure what you mean by virtualizing a raspberry pi. Are you trying to emulate its hardware, or just want to run a debian-ish OS on aarch64?

The DTB in your boot path is completely useless, because the DTB defines particular hardware to load kernel drivers for. You’re not running a VM customized with the same hardware as a rpi, so remove the DTB.

Like someone else mentioned, the kernel likely isn’t compiled with libvirt support and the qcow2 image isn’t using a GPT partition table and doesn’t have an EFI boot payload. You’ll need to grab something like armbian, or a different distro meant for aarch64 virtualization, like alpine. I believe Don from Novaspirit Tech (RIP) showed Arch Linux ARM (alarm) running on PiMox, as a VM. He might’ve left instructions on how to do that.

If you grab something like alpine and manage to boot it, you should be able to bootstrap a debian ARM image (even rpi os maybe), chroot into it and compile a kernel with the needed drivers for libvirt. Or also by booting alpine, you could grab the kernel, initramfs and /usr/lib/modules of a booting linux system (not alpine though, unless you have a sys type installation) and slap it on a rootfs of a different system (like rpi os).

1 Like