Data management advice needed

Software & Operating Systems Linux
1

Hi l1t!

I’ve started a PhD yesterday, and the laptop that will be my trusty companion for the next four years and me will need some advice!
First some system specs; I’m on a laptop with a single NVME SSD and am running Mint 20 cinnamon (with kernel 5.8 for better AMD support).

Through my university, we have access to some cloud provide that I can access through owncloud (which is pretty great, that they even have a linux accessible cloud).

What I want for my data management strategy is;

  1. have a copy of my data somewhere else (which is ensured by the cloud stuff)
  2. have regular backups to my personal cloud, also through owncloud

I want to request advice from you all for the second part. I don’t just want a copy of my data with a timestamp, I want it to only store changes (I don’t know the name of this, but something similar to how Git works).
Besides this, I want it to stay out of my way and just do its thing automatically and just give me access to my data, even in case I lose access to my computer, the university cloud stuff, or anything else.

Is there any software for this? Or how do I best approach this?
I’ve looked at Timeshift, but it seems to not be able to exclude /** and include /thefolder/I/want/** (it doesn’t matter in which order I store the rules). Everytime I tell it to include that folder, and then open the settings again, it is selected as exclude.

I hope some of you might be able to help me out!

2

maybe you could use git itself, as a private repo

or maybe there’s something like it that also works and I don’t know

3

Are you looking for something that doesn’t just take a full backup of all files every time, but is capable of only transferring the files that changed since the last backup? This would be called incremental backup.

Or are you looking for something that can backup what parts of a file changed, like git does with text files?

4

rsync is your friend.

For many years I had a rsnapshot (an rsync wrapper) based setup, where every night a cron job on my desktop would create incremental backups of itself and my laptop, and store it on a different physical drive (if you do not have a desktop, you can easily do this with a Raspberry Pi). The backup of the laptop was “optional” to accommodate for nights when I wasn’t home (say when I’m on a trip). Essentially I had to setup three things:

  1. key based SSH access between the machine that runs the backup and all the machines being backed up
  2. write .rsync-filter files in the top level directories of all the machines to control what is being backed up at the source; example:
    - *[Cc]ache
- Steam
+ /.bash_completion
+ /.bash_profile
+ /.bash_logout
+ /.bashrc
+ /bin/
+ /Pictures/
+ /code/
+ /.config/
- /.config/abrt
- /.config/google-chrome*
- /.config/transmission/
+ /.conkyrc
+ /Desktop/
+ /Documents/
+ /.emacs.d/
- /*
  3. configure rsnapshot and setup cron jobs that runs this periodically. I had yearly, monthly, weekly, and daily incremental backups.

The best part about this was I could experiment with several rsync based tools (step 3) before I arrived at my choice of rsnapshot. If you prefer keeping it simple, you could write a script and directly use rsync yourself, for an incremental backup you need to use a command like this:

$ rsync -aHAXF --log-file=/tmp/backup.txt --link-dest=/path/to/old/backup/ \
      /source/ /path/to/new/backup/

if you are a more graphical person, you might want to try BackupPCinstead.

1 Like
5

Hi, welcome to the forum :slight_smile:
I can recommend Borg as a nice piece of software that might allow fitting your needs.

6

Yeah, I’d recommend either Borg or Restic.

7

Thanks everyone for your replies and ideas!
I’ve looked at all options and there is a nice minimal gui for Borg, Vorta, thats in the mint repos, I will try that!