I’ve got about a dozen Debian LXC containers (all on Proxmox, but across 4 hosts, and 2 different independent locations) that keep filling their disks up with logs, and/because logrotate has not been running.
It appears this issue started when I upgraded them to Bullseye. From what I understand, this also is when Debian set logrotate up to use systemd timers to run. Looking at the cron file for logrotate, it literally ends if /run/systemd/system/ exists, I guess to prevent it running twice.
Upgrading to bookworm has not solved this issue.
Does anyone know where I should start looking? Should I just go with my gut and rip systemd out (again)? Or is there some quick fix like reinstalling logrotate that I could try first?
Interesting, probably something went wrong with the upgrade.
I don’t know what standard Debian does these days, I use journald for log storage, and I don’t think I even have syslog-ng/rsyslog/… traditional logging daemon or logrotate installed. Journald has a few options for managing disk space:
Nothing in the serverfault link really seemed useful - the status file lists dates in September and older on the random VM I just looked at.
The thing is… Logrotate works fine if I run it manually. It’s not logrotate itself that’s broken, but something to do with whatever is supposed to be running it.
I don’t know much about practicals of LXC, but enough otherwise to chime in. This is from POV of docker/containerd user.
Where are logsbeing generated and how is logrotate being informed that it should rotate them?
Logrotate requires explicit declaration what it should handle and how. I dont remeber whether wildcard catchall rules are possible, but likely they are (edit: its possible)
Verification needed on these points:
problematic log file locations ( ncdu is your friend here)
are there any rules and mechanism regarding their naming ( expectation is unique logfile per container or per each unique instance of container). If therer is catchall rule, they might be created with name and location that escapes it.
do logrotate policies for these files exist?
can lxc log output be changed to different backend? (journald …)
is in-container logging redirected to the host, or does it keep logging inside volatile container storage? If yes, then its container specific issue and must be handled by dev.
Easiest way would be redirect container log output to journald and let it handle it.
They’re all stock log files like syslog, kernel, auth, etc. Stock rules supplied by Debian exist, and work if run manually using the stock Debian logrotate.conf.
I’ve finally got somewhere now though… It seems that despite systemd list-timers showing “pass” for the job, it’s actually failing. systemctl status logrotate.service gave me this: logrotate.service: Failed to set up mount namespacing: /run/systemd/unit-root/proc: Permission denied
A github issue suggests my only options here are a security risk that I’d need to assess carefully per container, or just rip systemd out like I was already thinking I might do.