I’ve recently built a NAS that I want to use for backups of some folders on my PC. But because my PC has ext4 file system, and the NAS has btrfs raid1c3, the data on the NAS is certainly much better protected against bit rot. And to have some use for this protection, I want to synchronize the folders according to the following logic:
If a file in the local directory is newer than the one on the nas, copy from local to nas.
If the modification times match, but checksums do not, write an error to the log so that I can later resolve the conflict myself, or alternatively copy the file from nas to local.
You may want to use snapshoting feature at the filesystem level to provide version history. Hopefully directly on the NAS at scheduled interval or you can have Syncthing pass custom commands (not 100% sure on this one)
yet another +1 for syncthing. throw a wireguard (or tailscale) tunnel ontop and you can access the most recent version of your files anywhere at all times.
I use syncthing as well, both for home and at work because oftentimes I am my own IT.
Make sure you avoid cruft build up by adding the self cleaning after X days function of Syncthing (I have mine on 90 days) with the Simple File version copies (at 3-5 copies). It keeps drive usage and reliability at a good balance.
I’ve researched the syncthing tool, and I have a few concerns…
If I understood it correctly, it looks like if there is a conflict (which would be what it detects in case of bit rot), it just silently creates a renamed copy of a file. I don’t want my folders being cluttered with garbage by the synchronization tool, I want them to remain clean. This build-up of garbage is the reason I stopped using the dropbox client, and switched to FreeFileSync for cloud folder synchronization before I got the NAS. Is it possible for it to instead write a log file with the nature of the conflict, and leave the files unmodified until the conflict is resolved? Or better yet, a notification in case of a conflict, so that I can resolve it right as it happens? And better still, somehow configure it to resolve conflicts by itself according to the strategy I outlined in the OP?
I see that it’s possible to make folders “send only” and “receive only” to avoid garbage build-up, but then I don’t see how it would help me with bit rot detection if the most up-to-date version (that would be configured for send only) is the one that has bit rot potential, while the backup does not.
Another concern I have is that it looks like absolute overkill… It’s a complete cloud synchronization solution. And while I’m not opposed to using something like that if it solves my problem, I feel that when when the NAS is mounted via NFS protocol, something much simpler that works on local folders would do just as well. Especially considering I’m not opposed to triggering the synchronization process manually instead of having the tool constantly scan for changes.
I’ve created a file, copied it over to the NAS, modified the original, and changed the modification date back with touch -r. This doesn’t produce any output in the mismatch.log.
The tried and trusted tool for decades for this is rsync. It’s good and can do a lot for you and is also the basis for numerous mirrors across the internet.
But it certainly falls behind with modern storage like CoW and snapshots and isn’t the most performant tool, especially not on high-latency connections.
rsync is like IPsec, or SMB…support is usually a given for dedicated and related software. Common denominator. It works with everything after all.
I had my troubles with it with large file sizes and have rsync in use as a fallback solution whenever I can’t replicate via snapshots.