I know this has been discussed in multiple different places, many many times over, but I am trying to get a sense of if my experience is way beyond the norm.
I have been running Proxmox for 2 years now in a homelab setting, I have about a dozen or so VM’s considting of pfsense, homeassistant, truenas ubuntu server, some LTXC’s, a dozen docker containers, typical homelab type use case. I am running a pair of 980 (non pro’s) 500GB as my boot drive in ZFS mirror for proxmox and store the VM’s on the boot drives as well, so all of the proxmox logs + VM’s are writting to the same pair of 980’s.
That said, if I am reading this correctly, I have done 135 TB of writes in 2 years - this seems unphathamable. Am I reading this correctly? I am also not sure why I am seeing some data integrety errors, I should probably look into replacing this drive soon (or monitor to see if the number goes up? and if not, maybe just let it be?).
After seeing this, I went looking for solution and this morning I did set the Storage=volatile option in /etc/systemd/journald.conf hoping that will help going forward, but wasn’t comfortable modifying anythign beyond that as the other options seemed slightly more invasive/less begnin. I did come across a github page where someone has a mod to write all logs to RAM and then dump to disc at a set interval which seems like a potential plausible option, but I wanted to check in here first to see if this seems normal, or just totally crazy. 135 TB in 2 years to me seems crazy… Honestly even 12.4 TB reads seems impossible.
Proxmox writes constantly (every few seconds) to these 3 directories:
/var/lib/rrdcached
/var/lib/pve-manager
/var/lib/pve-cluster
I mount them as ramdisks and flush to disk every few hours and on shutdown. You also need to restore them on startup.
pve-manager and cluster contain settings for proxmox, rrdcached contains log data for creating the graphs in the webgui.
What is your solution to dealing with this? I have read lots of options over the past day, and I did end up doing the following to hopefully cut down on some of the logs.
systemctl disable pve-ha-crm pve-ha-lrm corosync
I came across a script that will set up the ramdisk and flush timer, but even tho I could read the script… I just don’t love downloading a script and running it on my hypervisor…
I came across a script that will set up the ramdisk and flush timer, but even tho I could read the script… I just don’t love downloading a script and running it on my hypervisor…
No way around that, read up and verify it line by line or be ready to read up and implement the same by yourself.
Goal here is modify default configuration of “appliance”, its not supposed to be done by hand, hence the complexity.
Regarding the other points:
disabling pve-ha-crm pve-ha-lrm corosync is doable, if you do not run HA proxmox cluster and do not plan on running one in future
135 TB of writes over 2 years is not potentially much for VM host, but --180 GB per day? That seems way too much.
verify its not generated by hosted vms first, then fuck around with hypervisor guts
try using iotop in cumulative mode running in each vm for about an hour, then same for host system
It looks incredibly simple, nothing hiding in there (unless I am just blind), but I am more worried about future interactions or issues it may cause. I am not a fan of modifying Proxmox itself, but this does seem like a slick little solution to me. Thoughts?
For what it’s worth, I went ahead and got the script running tonight. Won’t really be able to do a valid comparison, but theoretically my SSD’s should have much less wear from here forward, at least that’s the hope. I still have some nested ZFS going on (Proxmox is ZFS mirror, and truenas and pfsense boot are both ZFS as well), but I don’t think those will be horrible…
For anyone who may come along to this at some point in the future, it turns out pfsense (which I have booting via ZFS) was a huge contributor to the issues. I had ntop running which alone was half of the writes pfsense was generating, the other half was seemingly just standard pfsense logging. I was able to get pfsense to save its logs to RAM, and implimented this specific proxmox log 2 RAM implementation: pmxcfs-ram/pmxcfs-ram.sh at main · isasmendiagus/pmxcfs-ram · GitHub
Between those two things, it looks like I went from writting ~10 MB/s to my SSD’s to less than 1 MB/s. A full order of magnitude reduction, I will take it!
