I’m primarily a developer, not an admin, but in practice I end up administrating servers too. Most of the servers I’ve administrated have been internal to my organization, and used for purposes where a bit of downtime isn’t critical. That may change, so I’m coming here to ask a few questions related to administrating Linux servers where the general public are using the server constantly.
Specifically, I’m curious what the best practices are with respect to updates. My intuition is that it’s best to do regular security updates, and security updates only, and then do distro version updates on some sort of pre-planned schedule. Is that correct? Let’s say I’m using debian for the sake of argument (though recommendations are welcome): is there some set of modifications I can make to my sources.list to get security updates only? And should I apt upgrade a live server? I know these questions sound very basic - it’s funny, because I’ve been “administrating” non-critical servers for years but at soon as customers might be using the server at any given moment, I feel very uninformed.
Secondly, I’d love to hear what strategies the admins out there use for backing up live servers. I typically use ZFS and replication for the actual served data; it’s actually the root partition that I’m asking about. In the past I’ve used cron and rsync or whatever for this, but I recently became aware of lvm snapshots, which are appealing in the sense that I can get a consistent snapshot in time, and then back it up using any method as normal (as opposed to using rsync, where the underlying FS is potentially changing as the backup runs). Again, curious for input and advice.
in theory make a schedule, use APT pinning, and comment out or otherwise remove non-critical sources. use a kernel schedule and do maintenance around predefined time windows. “backups” can be an entire other can of worms. from VM clones and ZFS and RSYNC, then to the physical side of a safe in an off site location and rotating the disks on a schedule. this is all part of best practice.
so, as an admin in the middle of no where who is wildly under payed and under staffed, usually the rule is
if it is running DO NOT TOUCH IT.
if it was running and you did touch it, and now it is not running, YOU FIX IT.
if it is friday afternoon TOUCH ABSOLUTELY NOTHING!!!
@hsnyder
ZFS snapshots (and some kind of backup of configuration files) is usually a good idea. Not sure what the requirements are but in FreeBSD you can quite easily spin your own package repo using Poudreire which would make it very easy to keep it consistent between multiple servers and cherry-pick updates. Just run updates in crontab and you’re done.
Ah interesting! So you can manually test out updates on a server, and then if you’re happy with them, push them to your own package repo? I’ve used FreeBSD before and I’ve heard of Poudriere but I’ve never actually looked into it too closely.
Aptitude updates can be downloaded and/or applied automatically with the unattended-upgrades upgrades package. There are configuration options for security-only updates.
Depends on the machines being virtual in any way or if they are hardware. For virtual machines, there are tools/software to make daily (or more often, or less often) backups of the entire thing. In such a scheme, doing an “oops” will result in an adrenaline rush, a deep breath and rebooting the machine from the backup, job done, grab coffee. Not ideal since users may loose progress, but that is what maintenance windows are for.
Snapshots of any kind are super useful in case you need to work on risky settings.