I am really not sure where to turn here, so I hope some folks here can lend me a hand…
I have a homelab that has been stable for a few years now but I am now running into the strangest of issues. I run ESXi with ~4 Ubuntu LTSC 18.04’s and freenas with baremetal HDD access, along with a Win 10 LTSC for Veeam backup of the Ubuntu VM’s.
A few weeks ago, I would SSH into an Ubuntu VM and ZSH would prompt me of the file system being Read Only. I shrugged it off as a weird one time thing and restored from Veeam, but that it started happening more, and to more VM’s. Last night, sometime between when I went to sleep and woke up, all 4 VM’s had it happen…
Initially I thought maybe it was a Veeam backup durring an Ubuntu auto-security update (I do have that enabled), but Veeam did not run last night. I am restoring one of the smaller VM’s right now to an earlier state just to try and collect some info, but I am not even sure what to collect.
When I google the issue, it seems like the main answer is a mounted file system has errors, or FSTAB is the issue. I am not sure how this would be as one of the VM’s is not mounting any of my FreeNAS storage at all… It has no network links at all. I just got that VM restored (it just runs pihole… pretty simple VM, literally nothing to it at all except pihole), this is the fstab config:
UUID=2d945db3-1ff7-4c22-8022-0479207bb427 / ext4 defaults 0 0
/swap.img none swap sw 0 0
I run the VM’s (and ESXo) on a consumer SSD, and I upgraded to it maybe 5 months ago so its not exactly old, and its not like my VM’s hit it hard at all, its a very low use homelab, its on 24/7, but realistically its doing almost nothing 100% of the time. Could the VM’s be doing some sort of SMART check, seeing an issue with the SSD, and going to read-only as some sort of protection? How would I determine this? What else could be happening?