Making My Homelab Backups Boring, Testable, and Real
Well, apparently “maybe I’ll post updates here” has turned into at least one update.
The primary design goal was reliably boring. My ad hoc backup situation was a mix of application backups, NAS snapshots, cloud storage ideas, and the comforting belief that I would remember how all of it fit together if something broke. And I did not believe that last part.
I believed AI could help me achieve that: one documented, predictable package, created the same way every week, that I could test and still understand months later.
Naturally, the lazy thing to do was spend a few weeks in my spare time building a custom, application-level backup system so I would have less work to do if something ever failed. That is the kind of laziness I can get behind.
I am going to keep some of the implementation deliberately vague naturally. The interesting part here is the design and what I learned, not the names and locations of every moving part.
Lack of Copies is Not The Problem
Most of the important services already had something that could be called a backup.
There was application-native exports, NAS snapshots, infrastructure configuration backups, virtual-machine backups, and a few manual copies tucked away in places that made sense when I created them.
The problem was that these were all different kinds of protection.
A snapshot is not an off-site backup. A database dump is not useful if I cannot remember which configuration belongs with it. A copied ZIP is encouraging, but it does not prove the application can restore it. A full virtual-machine backup may be convenient, but it is not always the best or only recovery path.
I wanted one clear answer to the question, “Where do I start if I have to rebuild this?”
The answer I have now is the weekly bundle.
The Shape of the System
A dedicated backup service acts as the control point for the process.
The flow looks roughly like this:
Application-native backups and selected configs
↓
Local staging
↓
Verified weekly backup bundle
↓
Encrypted archive
↓
Separate off-site copy
A brute-force version would be easy: back up every VM and let the storage bill sort it out. I did not love the idea of paying to preserve copies of stock Debian installs. Application-level backups are smaller and more elegant, but the trade-off is fragility: I have to understand what each application actually needs to come back. This sounded like a very interesting project for a newly AI-enabled but lazy sysadmin.
Each of those artifacts are collected into a predictable staging structure. The weekly process checks that the required inputs exist, creates manifests and checksums, includes recovery notes, builds a compressed archive, encrypts the off-site copy, and verifies the resulting files.
Local and off-site storage have deliberately different jobs.
The local copies are convenient for quick recovery and keep a short rolling history. Local snapshots add protection against mistakes and accidental deletion. The off-site copy is for the much less convenient scenario where the local storage layer is gone.
Only encrypted bundles leave the house, and the decryption material is kept separately from the systems producing and storing those bundles.
There is also a separate longer-retention process. I chose not to automate every part of it. This is a personal homelab, not a bank, and missing one archival copy is not an incident requiring an executive review.
Bundle Contents
The bundle is meant to restore the lab’s brain, not clone every byte it has ever touched.
It includes things such as:
- application-native backups for important services;
- selected database exports and configuration;
- encrypted copies of sensitive recovery material where necessary;
- selected hypervisor and network configuration;
- manifests, checksums, and human-readable recovery notes.
I am backing up the parts I configured and care about, not paying to preserve another copy of stock Debian.
Disposable downloads, caches, temporary data, and large media libraries do not belong in the core weekly bundle. They may deserve their own storage or recovery strategy, but including them here would make the bundle heavier without making the important parts of the lab meaningfully easier to recover.
That decision made everything else simpler. I am backing up the parts that are difficult to recreate, not every byte just because it exists.
Untested Backups are Not Backups
Over my years, I have learned that these are three very different claims:
- “The backup job ran” means the job ran.
- “The checksum passed” means the file I have still matches the file I created.
- “The restore worked” means I used the backup to make something work again.
The scheduled weekly job is working is working now. It produces the expected local and encrypted archives, along with manifests, recovery notes, and verified checksums. The automatic off-site path has also completed successfully with an encrypted bundle.
One restore test during development reinforced the difference between a validated restore and a copy of some data. I restored an application’s built-in backup into a new temporary instance, but some needed environment-level settings were missing. Those settings now have their own encrypted backup instead of relying on the application’s native export.
Not everything benefits from being squeezed into the weekly process. Most of my Cloudflared tunnel configuration lives within Cloudflare itself, so the account is the source of truth. I keep a separate system backup as a convenient recovery path, but rebuilding the local connector would also be straightforward. My Smallstep certificate authority is local, private, and changes rarely, so creating a fresh encrypted archive after a meaningful change is more sensible than backing it up every week.
I now keep a small restore ledger as I go, distinguishing between checks and restore tests. That way, six months from now, I will know whether I proved something worked or merely remember feeling optimistic about it.
There is still more restore work to do, including periodically bringing an off-site bundle all the way back. A successful upload is useful evidence, but it is not proof that I can recover from it.
What I Learned
The biggest improvement came from deciding what not to back up.
Once I stopped treating every byte as equally important, the design became much easier. Some state deserves a weekly application backup. Some needs an encrypted configuration file beside it. Some is adequately protected by snapshots. Some changes so rarely that a manual archive is more honest. Some can simply be recreated.
The staging layer also turned out to be useful. Each service produces the best backup artifact it knows how to produce, and the weekly process packages those artifacts into one predictable recovery unit. When something is missing or stale, I can identify which part failed instead of debugging one enormous mystery job.
Keeping recovery notes inside the bundle was another small choice that felt important. If I am rebuilding after a failure, I do not want the only copy of the instructions to be in a system that may also be unavailable.
So that is where the homelab journey has taken me so far: not to a flashy new service, but to a pile of checksums, encrypted archives, restore notes, and evidence.
It is becoming boring.