Virtualizing Rpi on Ampere/Ubuntu 24.04 aarch64

I have a System76 Astra (ALTRAD8UD2-1L2Q) with lots of space and ram. None of that is doing me much good if the Rpi5 system images I want to virtualize with either give me a VGA or serial console. Of course, I cannot just boot a plain Rpi5 image, that would be #tooeasy. When enabling the VGA driver I just get the TianoCore boot shell :frowning: It does not see the qcow2 filesystem as a boot device. If I ls fs0: I see some files, but the TianoCore BIOS screen never lists a bootable file system.

Am I missing a .efi driver somewhere?

For console:

  • video=none
  • Copied the initrd and the kernel from the qcow2 file
  • defined the extra dtd
  • updated cmdline.txt to add console=ttyAMA0,115200
  • updated config.txt to add enable_uart=1

With some combinations I get a serial connection but no login prompt, but mostly I donā€™t get any sign of boot.

Console 1:

<console type="pty" tty="/dev/pts/6">
  <source path="/dev/pts/6"/>
  <target type="virtio" port="0"/>
  <alias name="console0"/>
</console>

Console VirtIO Serial 0:

<controller type="virtio-serial" index="0">
  <alias name="virtio-serial0"/>
  <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>

And Iā€™ve tried defining the dtb for the bcm2712-rpi-5-b:

  <os firmware="efi">
    <type arch="aarch64" machine="virt-8.2">hvm</type>
    <firmware>
      <feature enabled="yes" name="enrolled-keys"/>
      <feature enabled="yes" name="secure-boot"/>
    </firmware>
    <loader readonly="yes" type="pflash">/usr/share/AAVMF/AAVMF_CODE.ms.fd</loader>
    <nvram template="/usr/share/AAVMF/AAVMF_VARS.ms.fd">/var/lib/libvirt/qemu/nvram/2501-rpi5-build_VARS.fd</nvram>
    <kernel>/VMs/l_2501-rpi5-build/kernel_2712.img</kernel>
    <initrd>/VMs/l_2501-rpi5-build/initramfs_2712</initrd>
    <cmdline>rw earlyprintk loglevel=8 root=/dev/vda1 rootdelay=1 console=ttyAMA0,115200 rootfstype=ext4</cmdline>
    <dtb>/VMs/l_2501-rpi5-build/bcm2712-rpi-5-b.dtb</dtb>
    <boot dev="hd"/>
    <bootmenu enable="yes"/>
  </os>

Also, Iā€™ve been referencing Perplexity: https://www.perplexity.ai/search/i-am-running-an-arm64-server-i-QG9Ir_3iQUWrXgl9uBjVzw
Search results on S/O mostly seem to focus on amd64 based emulation (ok)ā€¦and after looking at so much of this Iā€™m not sure thereā€™s much of any difference by being on a fancy ARM CPU. #gottabeabetterway

Iā€™m pretty sure that kernel supplied with Raspberry Pi OS image is tailored for booting from device bootloader, not a tianocore uefi. If you have a real device - go check the kernel config it was built with. I wouldnā€™t be surprised if there are no VIRTIO or EFI related options selected. The machine type ā€œvirt-8.2ā€ isnā€™t emulating any of the devices the RPi image expects.

1 Like

That seems sound. I also found the Ubuntu aarch64 QEMU guide. I will work through that first.

I wonder if the Ubuntu blog post author will respond:

OK this blog post seems to be useful:
https://blog.jitendrapatro.me/emulating-aarch64arm64-with-qemu-part-1/#The-Basics

emulating aarch64 using machine=virt is stunningly slow. Just installing packages from Debian 12 netboot not only takes like 3+ hours, but also fails. :frowning:

Making progress! I have been following the blog.jitendrapatro.me guide.
I was able to start netboot and get a qcow2 file installed with a system image.

A large part of the problem is that the virt-manager interface is does not entirely correspond to what the qemu-system command arguments look like. Using the qemu-system-aarch64 command to get this booted, I could then start converting the next command into libvirtd xml:

Install.sh

#!/bin/bash
qemu-system-aarch64 \
    -machine virt-8.2 \
    -cpu neoverse-n1 \
    -smp 4,sockets=1,cores=4,threads=1 \
    -m 3072m \
    -initrd initrd.gz \
    -kernel linux \
    -drive file=2504-debian-12-10-root.qcow2,if=virtio \
    -nic user,model=e1000 \
    -nographic

You have to hard-kill the qemu-system-aarch64 process after reboot. Then copy the kernel and initrd out. I could then make this run.sh:

qemu-system-aarch64 \
    -machine virt-8.2 \
    -cpu neoverse-n1 \
    -smp 4,sockets=1,cores=4,threads=1 \
    -m 3072m \
    -initrd initrd.img-6.1.0-30-arm64 \
    -kernel vmlinuz-6.1.0-30-arm64 \
    -device virtio-scsi-device \
    -device scsi-hd,drive=d0 \
    -device scsi-hd,drive=d1 \
    -device scsi-hd,drive=d2 \
    -drive id=d0,if=none,index=1,file=2504-debian-12-10-root.qcow2 \
    -drive id=d1,if=none,index=2,file=2504-debian-12-20-swap.qcow2 \
    -drive id=d2,if=none,index=3,file=2504-debian-12-30-home.qcow2 \
    -nic user,model=e1000 \
    -nographic

At that point I could ask perplexity.ai to translate that into XML and then compare what virt-manager was crafting for me. Notably:

  • disks need to be SCSI
  • remove ACPI
  • remove EFI/UEFI declarations
  • remove TPM

Another important distinction is that the TianoCore BIOS does not know anything about VirtIO. No wonder none of my disks were seen! the TianoCore will see some SCSI disks, but it cannot do anything with them :frowning: Possibly because they are not GPT formatted. But because this needs to do a direct-kernel boot, no GRUB or /boot is necessary.

ā€¦and there I goā€¦numbly doing a fsck.ext4 fs over the wrong virtual disk :face_with_diagonal_mouth:

so use the UUIDs because I just learned that the /dev/sdX order isnā€™t stable (I should have known)