Things like Clonezilla tick a lot of boxes, but it is hard to automatically refresh the image.
Microsoft’s Disk2VHD is actually scriptable and how cool is it to boot a VM of your other workstation should something die…
Some decades back I setup https://fogproject.org/ for a large multi national org. The support desk workflow was a lot of work to understand what they really needed.
1 Like
I’ve heard of this from other admins I’ve worked with but never actually used it myself I might have to take a look at it.
I mostly just rsync the main folders of various things over to my backup server (fedora with zfs pool)
1 Like
I’ve been using restic on a full-disk basis for a while now, and it’s solid. I’ve been meaning to make a liveusb with my scripts for restoring a system from backup, but I’m not 100% confident in it yet.
I rsync my encrypted backup sets to a Rasberry Pi Zero at my brother’s house. Using ZeroTier so didn’t have to do any network configuration other than ask “what’s your guest SSID and password?”. Offsite backup for $50 and no recurring cost.
Robocopy > Bitlocker VHDx > OneDrive
I created my latest cloud backup solution to exploit the relatively new OneDrive feature of partial-diffing of large files.
I have a ~150GB Bitlocker encrypted VHDx in a folder that OneDrive watches. When it comes time to refresh my files, I locally mount the VHD and use robocopy to sync.
I unmount the VHD and eventually, OneDrive notices the change and starts uploading the difference.
It turns out that this isn’t a great solution, because OneDrive likes to shit the bed and get stuck in an infinite pending state.
Three times I have had to delete the cloud copy of my VHD, reset the local OneDrive client and wait for it to upload from scratch.
Meh.
Is this where Transmission would be ok to mention? Even as a linux noob i managed to install and use it.
Building a backup NAS of my Unraid is a priority at this point even if its less powerful the only thing I want is a solution that clones my Unraid maybe RAID? Still this one is a head scratcher for me if anyone has good ideas maybe share it. Currently have an mATX B450 and an Athlon 2200G.
1 Like
Restic > local repo > OneDrive
I had never heard of Restic but last week I had a play with it and it has won me over.
It is somewhat analogous to git with it’s snapshots and tagging workflow. It includes simple encryption too. I like it!
I have deleted my Bitlocker VHDs and have switched to Restic using a local repository. That local repo is in a folder that OneDrive watches.
OneDrive is still crappy and I will switch to an object storage providers such as Wasabi at a later date as it’s clear from the Restic docs that this is the recommended setup.
Here are some simple PowerShell scripts I wrote to automate backups.
I use fog as a local PXE image server, still haven’t figured out the automated imaging bit yet.
I back up to my TrueNAS using robocopy (OpenVPN split tunneling doesn’t play nice with my samba share on windows so that makes file history a non starter) then use the software in truenas to encrypt locally and push to backblaze b2, works fantastic, I have actually tested my recovery when I nuked my ZFS pool on accident and it went off without a hitch.
I think I have a pretty solid solution, of course mostly based around ZFS.
- My NAS (Debian 11 + ZoL) uses encrypted data sets and uses Sanoid to manage snapshots. These let me easily restore on those occasions when I
rma file or two I didn’t mean to. For my use, having a few days of hourly snapshots and 1 month worth of weekly snapshots is really all that’s needed. - I have a second low power NAS (Intel Atom Avoton, was my primary NAS long ago) at a friend’s house, also Debian 11 + ZoL. It is connected in a Wireguard mesh to my main NAS using Tailscale. I know Tailscale is a proprietary service, but for free and for the features it adds I think that’s OK; I could swap it out with vanilla Wireguard on demand if needed. My primary NAS uses Syncoid to replicate the ZFS snapshots nightly with the
-woption so that the data is sent encrypted and is never unencrypted on the other end. - For my really important data (taxes, personal photo collection, etc.) I also use Restic to back up to Backblaze B2. Of course, all encrypted so B2 can’t see any of my data. While this isn’t a 100% self-reliant option, it is my backup in case of a large geographical impacting disaster and it’s so cheap for how little data I backup to them that they only bill me every two months once I cross $0.50 owed.
Importantly, I don’t bother backing up virtual machine disks or anything like that, as all my servers are configured using infrastructure-as-code tools and I can restore them on demand with a few commands.
1 Like
My storage and backup overview
My data storage and backup solution go hand in hand. The high-level is that I centrally store or collect all important data on an on-site TrueNAS and subsequently back it up in 2 offsite locations: an off-site TrueNAS server and BackBlaze B2. Backing up TrueNAS is simple and automated, Therefore to make my backups simple, I do my best to store (or transfer/collect) all my data on the TrueNAS box.
How do I do this for each data type?
Family/Personal Photos & Videos
I have a Photo dataset/share on my NAS and store all photos there. I use a consistent folder structure as follows:
Photo
- 2021
-- 2021 05 08 John Doe's Birthday Party
--- Charles A6000
--- Charles iPhone
--- John iPhone
This is simple for traditional cameras, slightly more complicated for smartphone pictures. I have tried Nextcloud automatic photo upload and I did not like it very much. I have found that using Apple Photo’s Smart Albums to be a reasonable method. I have a smart album that collects all photos taken by the iPhone model of my own iPhone. I then use 2 tags Copied-to-NAS and Do-Not-Copy. For snaps that aren’t important, they get the ladder. For good pics, they get the former. The smart album filters for photos with neither tag - i.e., photos I have not yet organized. I open Photos on a Mac, and then select several photos, hold option and drag over to the NAS directory shown above. Holding option is critical for Apple Live photos as it exports both the short video and the still image to the NAS directory.
I also use the iCloud Photos Downloader to download all iCloud pictures to a separate non-organized NAS share. The idea here is simply a backup of iCloud. I prefer not to use the cloud as a “storage” location, but I am ok with using it as a “transfer method” from my iPhone to the NAS, not ideal but it works.
Personal Docs/Generic Files
I have a big NAS share for docs and personal files.
I use SyncThing to sync some files between my laptops/desktops and have added TrueNAS to this Sync group so that the files are included within TrueNAS backups.
I use Nextcloud for some files and use TrueNAS as the backend storage so yet again: included in backup
Media Server
I don’t backup anything from the media share. Maybe someday I will. I am comfortable with my current backup, but if I were to improve it- I would rather have 2 offsite TrueNAS servers than a higher capacity offsite server that has room for media files. These are reasonably easy to replace from the internet.
VM and Docker containers
VMs and Docker containers use TrueNAS as backend storage and therefore are included in the backups.
My containers use docker-compose, so the config is just a .yml file which itself is backed up.
Bitwarden
At this time, I do not self-host Bitwarden. Therefore, I use the Bitwarden app to export my vault to a directory on my NAS. This can be automated by installing the Bitwarden CLI and then using a cronjob to run the export command.
Joplin
I use Joplin for notes. I use TrueNAS as backend, however these are so important that I do perform a separate backup by using Joplin’s export feature and save the backup/export to the NAS. This can be automated with use of Joplin CLI and a cron job.
pfSense
pfSense has a great backup via SSH feature. Its very simple to run yet another cron job that SSHs into pfSense, creates a backup and then saves to a NAS directory.
MacOS - TimeMachine
I use TrueNAS as the TimeMachine disk, therefore (the theme continues) backup is included with TrueNAS backups.
Work in Progress:
Physical Linux Desktops
For my non-VM Linux boxes, I haven’t found a remarkable solution yet. The actual files are all stored via Nextcloud or Syncthing, but for my system itself including config files and packages- I don’t have a good method yet. I’ve been exploring options and I think I will setup BorgBackup to backup my /home, /etc and maybe /var
I use Google Workspace for my personal email. I have not found a proper backup solution here. I have heard remarkable things about Synology’s active backup for GSuite/Workspace. I do not have a Synology and do not wish to buy one in the near future but would consider one “one day”. They seem great, I am just partial to ZFS.
But if its the best way to backup Google Workspace, than I’d consider picking up a basic 2 bay and maybe just rsyncing it to TrueNAS. Maybe I need to check out Xpenology
TrueNAS config
This changes very in frequnetly, so for now, I simply manually download the config file to my desktop and use SyncThing to sync it across other computers and the NAS. This can’t be backed up to TrueNAS for obvious reasons, so I rely on SyncThing to sync it across all my devices. In a full disaster, I could pull it down from B2.
The new L1 series on “ultimate home server” is great.
1 Like
I run an Unraid server and a TrueNAS.
I investigated so many backup methods from Unraid and eventually decided that if I want to backup my data because I care about data integrity than I should simply use ZFS and backup via ZFS replication.
I build a really simple TrueNAS and use it as my NAS (except media-server storage). I use Unraid as media-server storage and as a VM/container host. I’d like to move to Proxmox at some point but have not launched that project.
I don’t have a good solution for Unraid, but I would investigate either TrueNAS, or another ZFS based NAS as I think it will be an easier and more reliable backup option than what Unraid can provide.
I could do something similar to yours by just building a separate system for “essential data”. Just upgrade the CPU and get 2 drives run TrueNAS or FreeNAS with ZFS. I love the idea of running a Proxmox server for Applications/VM’s.
This is how I started and then I realized I want to move fully to ZFS (except media-server). It is a great way to learn about ZFS and see if it works for you.
1 Like
As man here do I use restic to backup all of my important data (mostly documents and pictures as well as configs and Docker data).
My VMs are backed up to my btrfs RAID10 array on the same host since I don’t have another server running all the time and I don’t mind losing them. My OpnSense will backup to Nextcloud every day and my phone’s do the same with all the pictures.
The only thing that would bug me when I would lose my server is all the Movies and TV shows, but it’s just too much to back up.
I back up important files and configs mostly manually, every month or so for files including cycling through external drives for those backups. I do configs mostly after a major change or upgrade to keep them fresh… I’ve thought about an auto solution but im so used to it now I just do it on autopilot