Well, sometimes they wont boot at least. Let me explain.
I’ve got some EC2 instances running Amazon Linux 2 and 2023 which use XFS as their root volume. Occasionally, after several reboots, the instance refuses to boot and gives a kernel panic, because it cant find the volume specified by the root=<some-uuid> kernel parameter. Its happening on many instances, not just one.
[ 2.968517] VFS: Cannot open root device "UUID=3e24fa18-51ac-48d4-bb06-97d5b9bf54bb" or unknown-block(0,0): error -6
[ 2.977578] Please append a correct "root=" boot option; here are the available partitions:
[ 2.985798] 103:00001 68157440 nvme1n1
[ 2.985799] (driver?)
[ 2.993206] 103:00000 18874368 nvme0n1
[ 2.993208] (driver?)
[ 3.000416] 103:00002 18872303 nvme0n1p1 1a08294a-ac6f-4b04-874d-f320e8ca6fed
[ 3.000417]
[ 3.011913] 103:00003 1024 nvme0n1p128 b5bd0374-e403-4569-b87c-24908a66a5a3
Now this is where it gets weird. If I take the root volume from one of these instances, attach it to an intermediate EC2 instance, and (this is important) mount/unmount the filesystem, I can reattach it to the formerly broken EC2 instance and everything boots just fine.
Because its XFS, I assumed that there was some corruption happening. Instead of mounting it, I tried running xfs_repair. This tool refuses in every instance to repair the volume because there are log entries:
ALERT: The filesystem has valuable metadata changes in a log which is being
ignored because the -n option was used. Expect spurious inconsistencies
which may be resolved by first mounting the filesystem to replay the log.
Dropping the log with -L does not make the volume bootable again.
What should I be looking for? Is there some way to mount/unmount the root volume before shutting down? Any other ideas to track down what is actually happening?
I would appreciate any help. I’m here to ask the Elder Statesmen of Linux because AWS support promised to help, but stole my wallet instead
Could you post the full contents of the /etc/fstab too please?
Here is one example. I got all this data by taking a snapshot I have of a known bad instance, and mounting the volume to an intermediary instance.
UUIDs may not match the original log messages, as I copy pasted those from other notes I had while debugging. Ill grab system logs from AWS using this same bad snapshot as a root FS shortly.
Not directly, no. I have an AMI I baked, and am specifying a new root volume size when I launch these instances. Now that you mention it, I do suspect that AWS might be resizing the fs for me? I just never even questioned it.
I found this thread about someone who resized and it changed the UUID’s of their instance.
While it might work, the problem is that without the trip through an intermediary instance mount/unmount, I cannot even boot the instances in question to do this.
I think you are onto something. On an instance that isn’t affected, the UUIDs from lsblk -o +UUID and /etc/fstab match. On an instance that is affected, it seems like they don’t match, at least from the data I have here? Super weird. At least its a new lead?
Working theory is that something isn’t updating the UUID in one of these places… Turns out I had mounted the wrong device to explore. UUIDs match between fstab and lsblk output.
The Root Volume replacement stuff… at least I have never seen it work IRL
What happens before they reboot? Do they update the kernel and initramfs or bootloader? Is the virtual hardware changing?
First of all, both FS’s and block devices can have UUIDs which are parsed in userspace by udev, and show up in /dev/disk/by-uuid. XFS uses v4 UUIDs which are the same v4 UUID format as UEFI partition UUIDs, so be careful not to confuse them
Your lsblk output shows the UUID of the XFS filesystem itself, which looks good so far, but it also shows that the block device has a xvd kernel name prefix, which is used by the Xen paravirtual block device driver.
Your original dmesg output shows only nvme devices (could be virtualized or physical), so I suspect that the first time you boot it, it is booting from a Xen block device where the XFS root FS is, and the later time it boots, it has only nvme devices or the kernel is changed/updated to one without CONFIG_XEN_BLKDEV_FRONTEND / CONFIG_XEN_BLKDEV_BACKEND modules enabled/compiled-in or missing in the initramfs.
Yeah, I was getting a little loose with my terminology there.
Turns out the cloud-init automatic partition/volume expansion is failing occasionally (even on stock AL 2023 images!). Found some logs referencing xfs_growfs failing on \! This resize behavior was getting triggered on each reboot, too - I assume because the auto-expanding partition was leaving some blank space (in the repro case, a 1M chunk) at the end of the disk.
I fixed it by setting resizefs to false, and handling it a little differently in the bootcmd of the cloud-init.yaml I already provide. It lets me get a little more specific with how I resize the filesystem as I bring up the new EC2 instance.
