[Solved] XFS Root Volume Won't Boot

Well, sometimes they wont boot at least. Let me explain.

I’ve got some EC2 instances running Amazon Linux 2 and 2023 which use XFS as their root volume. Occasionally, after several reboots, the instance refuses to boot and gives a kernel panic, because it cant find the volume specified by the root=<some-uuid> kernel parameter. Its happening on many instances, not just one.

[    2.968517] VFS: Cannot open root device "UUID=3e24fa18-51ac-48d4-bb06-97d5b9bf54bb" or unknown-block(0,0): error -6
[    2.977578] Please append a correct "root=" boot option; here are the available partitions:
[    2.985798] 103:00001   68157440 nvme1n1 
[    2.985799]  (driver?)
[    2.993206] 103:00000   18874368 nvme0n1 
[    2.993208]  (driver?)
[    3.000416]   103:00002   18872303 nvme0n1p1 1a08294a-ac6f-4b04-874d-f320e8ca6fed
[    3.000417] 
[    3.011913]   103:00003       1024 nvme0n1p128 b5bd0374-e403-4569-b87c-24908a66a5a3

Now this is where it gets weird. If I take the root volume from one of these instances, attach it to an intermediate EC2 instance, and (this is important) mount/unmount the filesystem, I can reattach it to the formerly broken EC2 instance and everything boots just fine.

Because its XFS, I assumed that there was some corruption happening. Instead of mounting it, I tried running xfs_repair. This tool refuses in every instance to repair the volume because there are log entries:

ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used.  Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.

Dropping the log with -L does not make the volume bootable again.

What should I be looking for? Is there some way to mount/unmount the root volume before shutting down? Any other ideas to track down what is actually happening?

I would appreciate any help. I’m here to ask the Elder Statesmen of Linux because AWS support promised to help, but stole my wallet instead :smiley:

Could you post the full contents of the /etc/fstab too please?

And is the root volume mounted via iSCSI or is it dedicated block storage?

Could you post the full contents of the /etc/fstab too please?

Here is one example. I got all this data by taking a snapshot I have of a known bad instance, and mounting the volume to an intermediary instance.

UUIDs may not match the original log messages, as I copy pasted those from other notes I had while debugging. Ill grab system logs from AWS using this same bad snapshot as a root FS shortly.

/etc/fstab from a known example of a bad fs:

#
UUID=86f13586-c516-4493-81ea-e6f07cf45644     /           xfs    defaults,noatime  1   1
/dev/sdb        /earthly-cache  ext4    defaults,x-systemd.device-timeout=10,comment=cloudconfig        0       2

Edit: I had grabbed the wrong fstab, because I mounted the wrong device. This also blows up my later theory that there was a UUID mismatch.

For good measure, the corresponding disk from lsblk -o +UUID after mounting on an intermediary instance:

xvdf      202:80   0  18G  0 disk             
├─xvdf1   202:81   0  18G  0 part             86f13586-c516-4493-81ea-e6f07cf45644
└─xvdf128 259:2    0   1M  0 part        

The UUID on xvdf1 is what is in the kernel command line when it refuses to boot.

is the root volume mounted via iSCSI or is it dedicated block storage?

The root volume is an EBS volume.

Did you recently resize any of these shapes?

I found this thread about someone who resized and it changed the UUID’s of their instance.

The soluion they did was:

sudo mount -o remount nouuid /

And then they were able to correct the fstab entry to the new UUID from xvd

Did you recently resize any of these shapes?

Not directly, no. I have an AMI I baked, and am specifying a new root volume size when I launch these instances. Now that you mention it, I do suspect that AWS might be resizing the fs for me? I just never even questioned it.

I found this thread about someone who resized and it changed the UUID’s of their instance.

While it might work, the problem is that without the trip through an intermediary instance mount/unmount, I cannot even boot the instances in question to do this. :frowning:

I am familiarizing myself with the documentation

I think you are onto something. On an instance that isn’t affected, the UUIDs from lsblk -o +UUID and /etc/fstab match.
On an instance that is affected, it seems like they don’t match, at least from the data I have here? Super weird. At least its a new lead?

Working theory is that something isn’t updating the UUID in one of these places… Turns out I had mounted the wrong device to explore. UUIDs match between fstab and lsblk output.

The Root Volume replacement stuff… at least I have never seen it work IRL :smiley:

1 Like

XFS as their root volume

*root FS

after several reboots

What happens before they reboot? Do they update the kernel and initramfs or bootloader? Is the virtual hardware changing?

First of all, both FS’s and block devices can have UUIDs which are parsed in userspace by udev, and show up in /dev/disk/by-uuid. XFS uses v4 UUIDs which are the same v4 UUID format as UEFI partition UUIDs, so be careful not to confuse them :slight_smile:

Your lsblk output shows the UUID of the XFS filesystem itself, which looks good so far, but it also shows that the block device has a xvd kernel name prefix, which is used by the Xen paravirtual block device driver.

Your original dmesg output shows only nvme devices (could be virtualized or physical), so I suspect that the first time you boot it, it is booting from a Xen block device where the XFS root FS is, and the later time it boots, it has only nvme devices or the kernel is changed/updated to one without CONFIG_XEN_BLKDEV_FRONTEND / CONFIG_XEN_BLKDEV_BACKEND modules enabled/compiled-in or missing in the initramfs.

Yeah, I was getting a little loose with my terminology there.

Turns out the cloud-init automatic partition/volume expansion is failing occasionally (even on stock AL 2023 images!). Found some logs referencing xfs_growfs failing on \! This resize behavior was getting triggered on each reboot, too - I assume because the auto-expanding partition was leaving some blank space (in the repro case, a 1M chunk) at the end of the disk.

I fixed it by setting resizefs to false, and handling it a little differently in the bootcmd of the cloud-init.yaml I already provide. It lets me get a little more specific with how I resize the filesystem as I bring up the new EC2 instance.

Thanks for all your help @xzpfzxds and @Dynamic_Gravity!

2 Likes

Glad you got it working, nice troubleshooting! cheers