Hi All,
TLDR:
i need a fast and efficient cross platform selective file synchronisation solution that will work over my local network or the internet. it should fail gracefully on network loss and automatically reconnect and sync on network restoration. Servers containing all files need to have snapshots, that are also backed up offsite. Silently failing is not an option, so setup needs to support telegram / discord / push notifications / email. complex CLI / scripts for initial setup are no problem, but i need a GUI for management, monitoring, choosing what is synced where etc. once setup it should be very hands off.
have had some success with Resilo sync, but essentially it is too resource intensive and is quite slow even over the local network and i had some issues with cad software file locking resulting in files not getting synchronised until the software was closed.
Essay
Apologies for the essay. This is something i have been struggling with for years, so i am hoping the hive mind can help.
I use multiple devices for different tasks, some of these tasks and therefore data overlaps between devices depending on my location / convenience etc. Sometimes i will be in locations with no internet (gasp!) and no cellular, so require local copy’s of relevant data on my laptop / phone. Synchronisation should happen automatically over the local network or internet.
If it wasn’t for the cost and hard drive size restrictions i would want all my data, all the time, on all devices, with any sensitive data encrypted. For practical reasons i will settle for just having the data most relevant to that device synchronised. Some data would benefit from a one way sync such as Photos from phones.
On top of the synchronisation requirement i would like to follow a 3,2,1 backup system with snapshots to be resilient to accidental deletion, ransomware etc. some data would benefit from hourly snapshots, but most data would be fine with daily snapshots. ideally i would like to try and avoid single points of failure to fully satisfy the 2 in 3,2,1. one part of the backup software with a bug shouldn’t be able to take out all the backups.
For local backups / All data sync i have a old desktop running unraid with 2 x 8TB + parity and a synology ds 1512 with 2 x 8TB mirrored. i am open to suggestions for a offsite host, 2 - 4 Tb should do it as i would only keep the most recent snapshots offsite.
The solution needs to be cross platform. Win, Mac, Linux Android and I wouldn’t rule out iphones in the future. it should efficient and capable close to Gb/s transfer speeds and scalable to 10Gb/s transfer speeds when i upgrade my hardware.
I used Resilo sync in the past as combined with boxcryptor for sensitive data, it met almost all of the above requirements. some things it did well, like one-way synchronising photos from phones or synchronising small files.
However it was fairly resource intensive having a noticeable impact on powerful desktops / laptops and it brought the synology NAS to its knees even after letting it catch up for a few weeks. As soon as any significant amount of data needed to be synchronised the NAS would just be clicking away, 100% volume utilisation and transfer speeds in the kb/s resulting in everything loosing sync. i tried a load of different “fixes” but ultimately Resilo is not up to the task of dealing with large amounts of data on lower powered hardware. Even syncing between higher powered hardware i would be lucky to see double digit MB/s for more than the odd second.
what other solutions may i be missing?
Is there an Rsync based solution with a GUI and notifications that can work cross platform?
how to deal with locked files?
Thanks