How to setup FreeBSD on The Cavium Thunder X2 (Gigabyte R281-T91)


Cavium Thunder X2 – An ARM system that’s fast, power efficient and cheaper than the x86 equivalent? What’s not to love?

Well, it’s a bleeding-edge platform. It’s going to have some teething issues – though since this is a second generation we’ve seen much less issues.

I’ve successfully loaded (and prefer) RedHat but I thought I would give FreeBSD a spin to see how that works out.

IPMI Install

While the IPMI on the Gigabyte R281-T91 is great for installing most flavor of Linux, I’ve found it’s not the best choice for installation of FreeBSD. My experience with this was that up-to-date FreeBSD install builds are generally not geared toward ISOs and ISOs are what work best with the IPMI virtual media.

Further, windows users be warned. In putting this guide together, I tried several imaging solutions on windows. None of them worked. This machine expects an EFI image. If you aren’t familiar with what that means, it means that any bootable media you install had better have an ‘msdos’ formatted EFI partition that contains EFI binaries. If you image the USB disk from windows and insert it more than likely you’ll be seeing the “Protective MBR” partition. The problem is that this partition likely doesn’t match reality of what is on your USB disk. Usually that’s ok but not here.

The problem symptoms I had trying to get it to boot from these images was that the EFI shell did not detect the parition layout, and no EFI partition was accessible or selectable. If you are having trouble with the EFI boot menu and are on at least Bios F10 on the R281-T91, then the USB stick is probably imaged wrong.

I recommend good quality USB3 installation media. Some USB sticks interfere with partitioning if you try to setup more than one partition. I used cheap Microcenter branded USB sticks without issue. Lexar or Sandisk generally are good choices as well.


I downloaded the 13-CURRENT memstick image from the above link and used Ubuntu to dd the image:

sudo dd if=Downloads/filename.img of=/dev/block-device bs=10M

DD is extremely destructive, so be careful. if is the input file and of is the output destination. lsblk can help you find the input.

Again, if you’re on Windows, sorry. Forget it. Download a Linux live USB image and use that to setup FreeBSD. I don’t think anyone that uses FreeBSD has tried to setup an arm image from Windows; it’s not fun.

You might get a message like this:

pcib12: failed to allocate initial memory window:

I think FreeBSD should normally continue here, but it hangs for me on the 13-RELEASE version.

I rolled back to 12-RELEASE and get this:

panic: vm_fault_hold: fault on nofault entry, addr:

(to be continued)


Did you do a sync before removing the USB stick? I always use conv=sync with dd on Linux to make sure stuff is written to the disk not left to rot in a buffer. Not sure that’s the cause of the problem though.

Cavium ThunderX2 is a a tier 1 2 supported platform on FreeBSD to my knowledge , so this should work fairly smoothly though I can’t say I’ve had access to hardware to try it myself.

I’m not sure what kind of console you’re using but clearly the boot loader is using ansi escape sequences for colors when your terminal isn’t equipped to interpret them.

First thing’s first I’d figure out how to configure the console correctly. It might be set to a different console so you’re not seeing the kernel boot messages after the bootloader kicks it off. Are you looking at a serial port?

Might want to try an actual release image rather than a snapshot build :smiley:

Might want to try a more recent 13 snapshot and latest firmware :smiley:


Seems relevant:

Only amd64 and i386 are tier 1.

13-CURRENT has several fixes recently for ThunderX2 (on Sabre boards), which is why I recommended 13 snapshot images on twitter.

1 Like

The snapshot image of -13 is what hangs as in the first screenshot.

I was thinking the fixes were in as of a couple of weeks ago but perhaps that’s not the case.

Judging by the mailing list thread, you may need to build a custom kernel config with the NUMA option, but maybe that’s just for NUMA support :thinking:

Another set of changes needed for ThunderX2 support has been committed
over the past few weeks (thanks [email protected] for fixes and reviews). The
change were to: add IORT table support (r343853, r343856, r343860),
fix a crash seen with INVARIANTS off (r343875, r343876), and handle
more memory regions from the firmware (r343764).

I’ll translate some of the lingo for convenience.

The mentioned revisions correspond to the following reviews:

The last one there has a commit that looks like it should fix the panic in your screenshot:

The most recent 13.0 snapshot is tagged r344648 so that should have r343876 in it.

So I guess back to figuring out what the original boot issue was :sweat_smile:

The “Exiting menu!” message is from pressing 3 in the boot menu. At the ok prompt after exiting the boot menu you can continue the normal boot by entering the command: boot

Does the 12-RELEASE boot menu have the mess?

Yes. 12 has the mess but kernel panics after 1 sec. Even without exiting menu it just hangs at the memory map error.

The ANSI barf is simple enough to fix if you can’t configure the terminal itself to recognize them. Edit /boot/loader.conf to set loader_color=NO and it should use the black and white only text.

1 Like

There are two other images from the weeks prior to the one I linked, have you tried either of them?

Or there’s “build everything from source and call it a benchmark” :duck:

I imagine the generic arm64 FreeBSD image would run in a VM on Linux well enough to do a build :thinking:

Could try disabling that pci bridge by entering the following at the boot prompt:

set hint.pcib.12.disabled=“1”