Workaround for your server that won't boot from NVMe

Operating Systems & Open Source Linux
1

I’m sure this won’t be news to some of you Linux wizards but I wanted to share because it took me a while to figure this out, and I didn’t find all the pieces of the puzzle in one place.

I have a server that only boots from 2.5" drives in the storage bays or USB, but I’m passing through my storage controller to a VM so I was stuck booting my hypervisor from USB which isn’t great. I’d prefer my hypervisor’s main system drive to live on NVMe drives which I can install in PCIe slots via riser cards, but the machine won’t boot from them.

This workaround requires a basic USB stick for the bootloader and NVMe to USB adapter temporarily for the install.

My steps assume Proxmox configured with root-on-ZFS, which uses systemd-boot configured with proxmox-boot-tool, it should work with Grub or other operating systems but the details would be different.

  1. Use gdisk to put a GPT partition table on the USB stick with 1 partition of ef00 type (EFI system partition), this can happen before or after the Proxmox install or during the install using the Debug mode to access a shell
  2. Install Proxmox on your NVMe drive in a USB adapter
  3. Boot from the Proxmox system on the NVMe drive, installed in a USB port. If anybody knows how to skip this step that would be great because it would eliminate the need for an NVMe to USB adapter, unfortunately I was unable to do the next steps from the installer itself, and could not boot from the NVMe drive natively until next steps are done, hence the need to adapt the NVMe to USB…
  4. Run “proxmox-boot-tool format /dev/sdX” specifying the newly created EFI partition on the USB stick
  5. Run “proxmox-boot-tool init /dev/sdX” specifying the newly created EFI partition on the USB stick
  6. Shut down the server and move your NVMe drive from your USB adapter into your PCIe slot
  7. Boot from your USB stick

The end result is your cheap USB stick only holds the bootloader while every other part of the system like logging, VM storage, etc, is all on your NVMe drive(s) so you’re not wearing out the USB stick or limited by the speed of the USB bus or adapters.

My server has some weird limitations which required me to do this but pre-Haswell systems and some newer ones never supported NVMe boot in the first place so I think this might be useful for those too. I’m happy to hear feedback if there is an easier way to do this or if there is a more general way that does not rely on the proxmox-boot-tool!

2

Proper servers support SATA-DOM (disk-on-module) which is essentially what you describe but with a small SATA disk instead. Low capacity used units are not expensive:
https://www.aliexpress.com/item/1005003943149016.html
No need for USB adapters, just point the installer to the NVMe drive and let the bootloader reside on the SATA-DOM drive.

3

That’s a good point!

Now, my “proper” server (Cisco C240 M5) does support something even better than that, they make a riser for m.2 SATA or NVMe drives mounted on the motherboard or a backplane for u.3 drives in rear bays, all of those would be bootable and on a separate controller from the main storage so they would be great as hypervisor boot drives, but I don’t have either of those risers.

Unfortunately the parts are all $100 or for the rear bay backplane, $100 + $150 shipping from Europe, and some of the other options are completely unavailable as far as I can tell.

It does have an internal USB port, which isn’t as nice as a SATA DOM, but adequate for just a bootloader.

I was using USB-to-NVMe sleds in the external USB ports before I figured out this solution, at least those wouldn’t wear out like a cheap USB stick, but they were slow and ugly and prone to damage sticking out the back of the server.