Server backup softwares and backup hosts

mihalycsaba · April 9, 2023, 2:47pm

Hello, I wanted to start a thread about server backup solutions and your experiences with them. I’m managing quite a few centos 7 servers, and data loss is always on my mind, just didn’t have time or possibility to try out more solutions. It would be a big problem for me, if I somehow lost data, literally the worst thing I can imagine . Curious if borg is a good choice.

Tried some commercial solutions(synology’s active backup, r1soft, synconix), they were all horrible and with serious bugs, not to mention expensive.

I’ve been using borg with borgbase as host. It’s fast and didn’t have issues with retrieving files from backup, the few times I’ve needed them. The host is really cheap, 12€ a year.

I know about Hetzner’s storage box, it’s compatible with a lot protocols, that’s next on my list to try as host.

I also use rclone with storj to store a local borg backup for a personal vps.

My favorite things about borg and necessary criterias for other software:

no need to open ports and run some service on the client machine for it
encryption
differential (maybe incremental, not sure if it’s a good idea with more than years worth of daily backups)
compression
cheap hosting available
saving unix permissions
corruption resistant (afaik borg is, if you verify periodically)

Did you have problems with borg?

Do you guys have experience with other software, that more or less fit the above list? I looked at a few, but I can’t decide without trying them out for a few weeks each. Contenders right now are restic, duplicity, duplicati, rsnapshot with cryptshot(rsnapshot inside luks)

Extra:
Do you think about disaster recovery? How would you go about it in case of physical servers and virtual too. I know about rear, just not sure how to save recovery images efficiently or make image from existing borg backup. Also I’m not sure if I could convince my vps host to run a recovery image for me. So I would like something that might be able to restore server to a previous state if I have at least ssh or command line access(novnc).

peppy · April 9, 2023, 3:43pm

I don’t have too much to add about Borg or the other “contenders” that you listed out, but my main offsite backup solution is using Google Drive. Mainly my partner already pays for 2TB of backup storage with them and they’ve graciously given me half for backups.

There’s cli tools like gdrive ( GitHub - prasmussen/gdrive: Google Drive CLI Client ) that have worked for my use case (plus the code is in Go, which is my main language).

According to /r/datahoarder’s wiki ( https://www.reddit.com/r/DataHoarder/wiki/backups/ and ctrl-f for “google drive” ) you can still get unlimited storage through Google’s Enterprise offerings for about $20/month. However they supposedly cap you at 750GB/day upload and 10TB/day download. Though I’ve not validated that myself. Because for my use cases (running a bunch of web apps for scraping data, media streaming, and infrastructure as code), my little 1TB space of backup has been plenty for me.

Absolutely!

But my services are all containerized (using k8s). So being able to test restoring from backup is as painless as just setting up another pod to run the service, running the script that I make for each service to restore data, and validating it works. I’ve been thinking about how to automate more of the testing aspect of the restoration (beyond just checking hashes of backup’d material), but I really don’t want to become an QA engineer during my free time

I’m still on my journey for making redundant infrastructure (literally just needing to buy the hardware). But once I do, then I can also treat the underlying infrastructure as replaceable at any time and truly lean into more of a “chaos engineering” approach for restoration and validation of making sure services are available.

Good luck in whatever you do choose for backups.

mathew2214 · April 9, 2023, 4:47pm

on my small scale system i manage, i have created bash scripts and cron job to make a RO BTRFS snapshot of the file-system, and then to rsync that snapshot to the backup storage array, then delete the snapshot. very convenient to be able to rsync a running system this way.

MikeGrok · April 10, 2023, 2:08am

Zfs snapshots
snapshot diffs stored locally and streamed to remote.
After a week remote applies them to the archive.

My special fear is corrupting the zfs dataset in a way that causes a loss of the dataset and its backups. I know someone it happened to twice in around 2010.

I know many of those bugs have been fixed, but it is annoying to have to wait years hoping that the next version of zfs will be able to import your corrupted zfs dataset.

rcxb · April 10, 2023, 2:35am

Borg has proven quite reliable. Its main problem is performance, being single-threaded it can be a severe bottleneck on large data sets. But if it is working for you, great! It’s also vulnerable to an attacker intentionally corrupting your backups rendering them unusable, but that’s true for most backup solutions.

ChrisA · April 10, 2023, 6:50am

I think my method is a simple one - playing with kiddie so took the opportunity to do basic schematic

Forgot to mention, these are 4 TrueNAS Core machines and I’m talking about replications.

risk · April 10, 2023, 9:15am

Backup strategy for my home router (which is a smōl PC with a Debian install with some network interface rules and some routing rules and some containers which have a lot of config and a little bit of data, all of it “secret stuff” like various certs and so on):

disk layout is: (/boot/efi, fat32) (/boot, btrfs) (luks2) ; within luks volume is btrfs with an @root subvolume and a few others for /var/cache and /var/log.
systemd triggers snapper every hour-ish and makes a btrfs snapshot of @root subvolume, btrfs snapshots are not recursive.
I wrote a small shell script that runs before that and rsyncs /boot and /boot/efi to /boot_extra
I wrote another script that runs after, it looks at snapper snapshots and invokes btrfs send | rclone cat . The script is a bit smarter, it actually sends incrementals, unless the newest full snapshot is too old.

My restore strategy, which I used once a few months back, and probably once before but can’t clearly remember, involves a laptop and a USB key, some partitioning, rclone configs, btrfs recv, rsync, chroot and reboot.

Overall it works but I’m NOT HAPPY with it, it’s all just BS half measures

The smòl router PC doesn’t really have a permanently attached keyboard or monitor, it has no ipmi or pikvm or network boot by default, it’s a PITA to restore, I have no idea whether my script/restore procedure really works.
How do you pick a snapshot to restore from – e.g. if it just stops working on its own, when was it things went wrong, which snapshot should I start with?

I think I learned my lesson, backing up configs or whole systems “for quick recovery”, is not the way to go. Instead, I should’ve built a redeploy script, and tests for it in qemu.

mihalycsaba · April 10, 2023, 1:38pm

Thanks for the replies, sadly the only common denominator in those servers is the OS, different physical servers, virtual servers with different virtualizations, different networks, different filesystems, but mostly ext4. Most of them weren’t even set up by me.

So yeah, filesystem snapshots(have to look into those though) or complex individualized for every system backup scripts are not a good solution for me, too much time to set them up properly.

Next year I will have to reinstall most of them, because centos 7 won’t be supported anymore. I will have to look into snapshots and backuping them somewhere.

I’m interested in only hosted solutions, don’t want to maintain a physical server somewhere for backups.

Anyway my theory is that you need at least two backup locations and technologies running parallel, you never know which one will fail for whatever reason when you really need it.

I’m going to try restic with hetzner as my secondary solution next.

@peppy last year I was storing my git repos in a veracrypt container, the first backup host I tried was google drive. The container’s size was at least 1 gigabyte. I downloaded it on another machine, but couldn’t open it, got corrupted in the cloud somehow. The local file worked correctly. So I wouldn’t trust google with files other than regular everyday photos and documents.

@risk redeploy style system backup is not the worst idea, especially for disaster recovery, maybe even more error proof than other disaster backup solutions.

I had two disaster situation and had r1soft(literally garbage) backup, it just didn’t work how it should have in theory. I’m also using timeshift for my personal workstation, it works great, if you don’t have to restore to a reinstalled system with slightly different setup.(maybe I just didn’t know how)

Cherry picking a minimal amount of configs that you think you will need to restore to a fresh install, could work. Though might take a bit of time to collect everything you will need. I might do this and test in qemu. Treating data and configs differently could be a good solution for me
Also there are some free monitoring services, that can send you email or whatnot if your backup doesn’t work. Monitoring is important.

mihalycsaba · April 10, 2023, 1:39pm

Isn’t backblaze slow when you want to download from it?

vic · April 10, 2023, 2:25pm

Home servers are pretty much idle 99% of the time. So the backup strategy might not apply to servers doing thousands or millions of transactions per second.

My backup strategy is very simple that try to keep two copies of the data (OS or user; ‘try’ is the key as the two copies are out of sync most of the time. lol). I also used to send a copy of important files to Google Drive by using Duplicati. But I no longer do online or off-site copy. I don’t mean to be of good practices. I just feel fun talking about my backup strategy:

I have an offline ZFS pool in a USB enclosure for backup purpose. ZFS is chosen because of 1) user data checksum 2) HDD/data scrub function 3) snapshot 4) concept of datasets 5) compression & encryption. I’m pulling these from my notes when I did the ZFS on-boarding.

I believe I’m an outlier the way I use ZFS: always availability of correct data is not my goal of using ZFS.

Besides the ZFS pool, I have a set of scripts and systemd services making use of rsync and ZFS snapshot to drive the backup. I called my solution TimeClone.

The standard operating procedures will be like this: when I feel necessary (often a few days apart), I plug in the USB cable of the ZFS pool. From there on everything is automated and orchestrated by TimeClone running on the host that the ZFS pool is attached to. It’ll rsync OS data, user data & etc from various sources and hosts. For MacOS, Apple’s Time Machine will auto detect and back it up to the ZFS pool. I usually (export and then) unplug the ZFS pool when it’s done. If the ZFS pool is kept plugged in, then TimeClone will do a daily backup for me from all the reachable sources.

Linux OS disk is perhaps the easiest to backup. I did disaster recovery once by using TimeClone restore. On another occasion I used TimeClone backup & restore in succession to transfer an existing Linux installation to a new drive.

It works well for me.

mihalycsaba · April 10, 2023, 2:40pm

This plugging in a USB sounds convenient, maybe you should publish this solution I have clients where this could work.

ChrisA · April 10, 2023, 3:10pm

I can’t remember the speed when testing, I do recall it saturating my 70mb connection though. Worth noting you pay extra to download.

Realistically speaking,if I was depending on it, something must have happened to both my house AND detached outbuilding. So I would have far worse problems than slow download, I might even take up their offer of delivering me my data on a BackBlaze portable NAS.

mihalycsaba · April 10, 2023, 5:29pm

How much do you pay usually? From the website it looks like it costs 6$/TB if you don’t download. Not too expensive.
But Hetzner is cheaper with no traffic costs afaik. There are free snapshots too. But only has servers in Germany and Finland. I’m gonna try it next week, hope it’s not too slow or something.

shadragon · April 10, 2023, 6:13pm

Veeam Backup and Replication has a free version (up to 10 machines) so you can give it a decent workout before buying. Windows and Linux image level backups for VM, physical or Cloud machines. File level or VM restorations and works flawlessly on both my home lab and in our four datacenter vSphere production environment.

Works with Cloud storage, NAS or a myriad of other storage. I have no affiliation with Veeam beyond using it daily and do recommend it.

Janos · April 10, 2023, 6:42pm

Veeam is so far my go to Backup solution, never had an issue with it.
At the moment I am also testing urbackup, so far mixed results. It works with Windows, Linux and Freebsd, but I had problems with rolling release Linux distributions

ChrisA · April 10, 2023, 7:36pm

It’s a long story that I’ll shorten.

I have around 18TB that I would like to backup using BackBlaze, but that would cost me the equivalent of a new and fairly decent computer every year (not including GPU). So instead I selectively backup (based on file extension) around 1TB, which is then around $60 per year.

Using the replication facility that TrueNAS has, I replicate datasets to 2 other machines. If my main machine were to simply disappear, I could turn read only to off on an unaffected machine and carry on as if nothing happened. I would of course repair the one that’s stopped working!

For me BackBlaze is really for an “everything else has failed” scenario and it’s very easy to setup in TrueNAS. There are some gotchas though, such as the necessity to use “Use-fast-list”, otherwise extra costs can be incurred (and big ones). There’s no cost with the upload though, just the amount of data stored.

I’ll certainly look into Hetzner though, but only if it’s easy to configure in TrueNAS.

mihalycsaba · April 11, 2023, 6:14am

Jesus Christ, the pricing is per machine… The daily modified data on some of the servers is minimal, sometimes a few dozen megabytes.

I entered 20 servers, 1500€ per year🙃That’s too much for my ‘market’. If I make an account for every company I manage, would managing/monitoring the backups be a juggling game? Do I need 20 different email addresses for myself to be able to log in and receive email notifications?

shadragon · April 11, 2023, 1:50pm

This was all I had to go on, no budget was mentioned. You wanted bullet-proof and Veeam has never failed me.

Daily, it’s my primary job. I live in DR ‘what-if’ mode all the time. With 360 Production / DevOps VM’s as my responsibility I need to be able to recover from any loss, up to and including an entire country going dark. Or reach back into the past, up to eight years, for emails or file information on a terminated employee for a legal case. At the moment, we hold around 260TB of backups. I’m sure it’s more than that.

Then what is your budget? You also say you are managing multiple company (20?) backups so you are deep into Enterprise level there as well. If you want a Veeam / Zerto level of protection, you need to pay for it. Hate to be that guy, but simple replication won’t cover you in the scenario you outlined. You need Point-in-time backups going back to whatever your stated retention period is in your engagement contract. If just doing data backup, you may be able to get away with it with Open Source, but good luck if you hit a snag and need support. Otherwise, you had better have the most patient customers (and the best written legal contract) the world has ever seen.

Cheers.

mith · April 11, 2023, 3:22pm

I highly recommend restic. I’ve been using this as a part of a layered backup strategy for a couple of years.

Here’s is what I do for my home servers:

Primary NAS is ZFS raidz2. I’ve never lost any data w/ ZFS due to drive loss. I’ve been using ZFS for almost 15 years. As long as you periodically scrub and keep up with drive replacements it will serve you well. I also sometimes use zfs snapshots. But since raid is not backup, I have two other tiers.
The second tier is restic. NAS runs a scheduled daily restic backup to a remote chromebox. The chromebox has several attached usb drives. I run the restic server on the chromebox as its faster than alternatives like FTP, SMB, or NFS. This tier is my snapshot tier. It enables quick, local restores of deleted or missing files. I run a weekly script on the NAS to prune restic snapshots according to my desired retention schedule. I like using restic here because its design allowed me to turn an old mini-pc into a low-powered backup server as the client does most of the work. Restic also performs well and has been rock solid in the approx. 2 years I have used.
NAS also runs the linux idrive client for an off-site cloud backup of my most important, never-lose data. Idrive is inexpensive relative to the other cloud providers. You could also use restic to send to backbaze, for example. Restic supports rclone as a backend, and that affords you a large number of alternatives for cloud storage.

mihalycsaba · April 11, 2023, 8:27pm

It’s like 10 times less. These veeam prices are made for a western market I think. Not many Romanian companies are willing to pay for data safety. Aprox. half of my clients will pay whatever I tell them(veeam would still be too much), but there are the other half, where I basically pay for the backup out of my pocket. Hell, some of them still use windows xp, because it’s still good for the data entry task it’s been good for the last 20 years.

So yeah I’m doing this for my peace of mind mostly. Most people here never think about their data, they can’t even begin to imagine what happens if it disappears. I don’t care if their data gets stolen, it’s not valuble data, they just work a lot with it. But losing everything could be a major pain in my ass for sure.