Veeam
I still need to read all posts on this thread. I have no idea what I’m doing so this is not advice. I have a PVE box and a PBS box. I intend to locate PBS offsite eventually as the nightly diffs should be small enough that upload/internet would be fine. For onsite backups I have a ZFS array in the PVE box and a directory set up on there with PVE doing a backup of all VMs nightly to that internal array, and then to PBS.
I was annoyed it seemed like I had to fight and bushwhack a bit to get that backup happening in the PVE box - seemed like they wanted a PBS box onsite and a second PBS offsite for replication. Maybe that would be better idk.
Just today (before seeing this post) I had the notion of sticking PBS in a VM on PVE… bunch of tabs open but have not yet investigated the internet wisdom on whether that makes any sense to do or not.
Will be interested to hear Wendell’s “strong opinions” as my stuff is not production yet but it aims to be once I’m more comfortable with it.
What we have been doing for our physical Windows Servers is the inbuilt server backup to usb HDDs that are exchanged weekly, one kept offsite. This is manual and kinda sucks. When we started using BitLocker encryption with these USB drives it got less bulletproof reliable, and there’s no good alert/reporting for it, and only defense against ransomware is to go back to last-week’s drive that’s been airgapped. The hope with using ProxMox to with the Servers as VMs is that ProxMox can ensure the backups happen and get replicated as needed in a reliable fashion. Being able to consolidate to fewer boxes is a small nice bonus.
oh joy, one of the hdds hiccupped or something and after a reboot zfs import is failing, even though the disks are all there. Maybe their ids have been scrambled? sda,sdb, etc?
1 Like
Backups are quite latency sensitive, so you really want them done onsite (or near enough). Replicating backups is fine over high latency links.
Many enterprise backup programs like Veeam solve this by having local proxies you can deploy to assist with this, but PBS doesn’t have anything like that (yet?).
Depends on a couple of things:
- the backup software you’re using (some are more sensitive than others)
- if you’re using compression and dedup client-side before sending the data to your backup server
- the date you’re backing up (DBs are more prone to I/O write errors when the backup server has high latency, while backing up files or blocks is usually more forgiving)
But I agree. Ideally one would want a local backup server (and if possible a dedicated non-routable backup VLAN) and then send / replicate the data off-site (particularly if you use zfs-send for replication).
I still stand by my opinion that you should only back up your most important data and have a good recovery strategy (preferably deploying fresh VMs). There’s a couple of scenarios (that I can think of) where it’d make sense to run snapshots on an entire VM:
- you’re testing stuff and want a quick way to revert in case something goes wrong (like updates) and that doesn’t require copying the snapshot somewhere else (i.e. not a backup)
- you’re mounting the snapshot on a different VM and using that to back up the contents (to save CPU and RAM on the production VMs - this is particularly useful if you’re backing up the rootfs, although not necessary)
- your software is really OS specific and you can’t quickly and easily set up a new OS with your previous software (this is particularly important for old and legacy stuff, like mysql 5.5 or php 7 and you can’t upgrade because of incompatibilities - good grief, please plan to replace that software, if you’re using such deprecated software, I’d bet you’re probably running it on centos 6.10 or something)
- you have really large workloads, especially large DBs (block-level backups are faster to perform, just lock your DB, take a snapshot, release the lock, then zfs-send the snapshot - this is way faster than doing a logical dump)
There’s probably other reasons, which I can’t think of right now. But in general and for most people, aim to only back up your necessary data alone. That’ll save space on your backup storage (which you need if you want to maintain multiple previous copies of your data, which you should). Don’t just back up your entire VM, OS and all. You can get your OS back in minutes with a fresh install (compared to if you were to recover it) by copying the data from backups. Just my $0.02.
Still seems odd to me that the preferred primary backup isn’t built into proxmox itself. I know you can just stick a drive in it and point backups to that, but my instinct was to have a zfs array for backups, and had to do some slightly off-script things to get that working, and at the moment it’s not “replicating” those backups to PBS but doing an additional backup to PBS. I thought having the backup happen inside the PVE box would get it done the fastest, minimal latency, as opposed to being across the LAN (1Gb/s at the moment). But perhaps I’m forcing a bad solution. Certainly I don’t love what I have at the moment.
I just YOLO it, vzdump once per day, dump databases every hour.
It’s not very space efficient, but it works for me since I don’t have huge VMs.
I am using PBS as VM in production SMB setup.
We have 3 node PVE cluster (one of them is basically just a voting node on Intel NUC mini PC to achieve HA).
PVE1 is the main node with NVME storage running 3 VMs one of witch is a working horse Windows Server VM with SQL applications.
PVE2 also has NVME storage where PVE1 is replicating VM disks to for 15 min. HA. Also runs PBS VM with passed through SATA controller+spinning rust storage.
PBS works very well as target for PVE1 VMs with high deduplication factor. Backup verification, garbage collection are all good stuff I like in PBS.
Additionally we backup VM’s to Synology NFS share and SQL DB’s separately with Synology snapshots and Synology C2 cloud.
I feel this is cost effective and good enough solution for small business 
2 Likes
Yeah i wondered about putting PBS in a VM inside the PVE box, to avoid a second physical server, but it kinda feels like (since I only have the one PVE node) it may be best to have them independent so they could tattle on each other.
i now have 2 PBSs, both onsite at the moment but one will be offsite soon. I suppose the thing to do is to have the onsite PBS trickle replicate over to the offsite one, since internet upload bandwidth is kinda slow.
Yeah, ideally if you can run a full sync now and ship it after. When it’s in its designated site, then you can run incremental sync. I’ve run a full zfs sync over the internet and boy it wasn’t fun (weeks download time).
1 Like