I wrote a script that backs up / into my $HOME/.backups/sys every day. It’ll first save a list of all package versions installed in a plain text format, and then it builds an --exclude-from file, for use with tar.
On my arch systems it uses pacman -Qkk to get a list of files to exclude, on my Debian systems it uses debsums. I also pre-seed the exclude lists to not back up some boring always changing files, like systemd journals.
The tricky part has always been restores, each one of my systems is different, most of them I haven’t had to restore. Some that I did it was a pain – it took a day or two.
For home dir backups, I use rsync to encfs sshfs. Most of my large media lives outside of my home dir, that’s ether not backed up or has a separate rsync
I have 26 HDD’s with MANY partitions at home… This blog wiil be a tale of a broken man, a sad story and a redemption at the end.
What is your goal with partitioning, maybe I can help without the video?! It sounds like your close??? Walk back your steps?
Well… I’m sure you can relate but I suffer from a programmer disease Lazy programmer-itis and over the years you write scripts to solve a quick problem, maybe troubleshoot/test it once or twice, then “The kitchen is burning cause you left a Pizza in the oven” and when you come back you forget, run the script and carry on. 10yrs later you wonder… “Why did I write this like this, why is this not here or organized better?!” It happens. . .
Thanks for the advice, I did finally get virtualized hard drive mounted and have the permissions set up the way I want them. I fixed the last problem I had by using nano in the terminal instead of Xed with root privileges. Now to test if the virtualized hard drive will behave like I think it will.
Just a quick progress report, I add a few movies from my Movies collection to the virtual machine installed Emby Server, put some movies virtualized new hard drive, tried accessing previously mentioned movies from my Samsung Smart Tv. Success, it works. So I do have the permissions set up on
the virtualized new hard drive correctly. Now the only thing left to do is to shut down my desktop, physically install a new hard drive and set up the new drive. Finally, I will be able to have my movie collection on one large hard drive instead of several. Thanks, everyone especially Magnus_Python.
OK guys. I must be doing something wrong. Here’s what I’m trying to do:
I have 10TB of media in the array /disks/PlexMedia
and a fresh copy of that array on another array /disks/PlexBackup
When I used Robocopy/MIR it took about 1 minute to make sure they were synced together if I hadn’t changed anything because it used the archive bit I believe.
I’ve tried many different options with both RSYNC and RCLONE and the best time I got was 30 minutes to complete on two directory structures that were the same.
Everything I tried on RSYNC ended up copying everything anyway(10TB takes FOREVER), even with the -u (update) option set. Can I get a working command that will copy any new/changed items from /disks/PlexMedia to /disks/PlexBackup and then remove anything from /disks/PlexBackup that no longer exists on /disks/PlexMedia?
Wrapped in a script, I use rsync to mirror a network share (which is what it sounds like you want to do) that is 28 TB. It takes about 20 minutes to run if there are not many changes:
where ${LOG_FILE} expands to /mnt/md0/mirror-log/rsync-2020-02-16.log
Snapshots
I also use rsync to take daily snapshots of the same network share. The rsync command is mostly the same:
rsync --log-file "${LOG_FILE}" -a --link-dest "${current}" "ryzen-server.danger-rocket.com::raid" "${target}"
where {current} is a symlink to yesterday's snapshot and {target} points to today’s directory. When that is done, the script updates the symlink to point to today’s snapshot, so it becomes tomorrow’s reference. Files that are unchanged are just hardlinks to the original.
That also takes about 20 minutes to run if there aren’t many changes.
If you find rsync is copying files you think are identical, check the log. The log file will tell you what changes rsync made, and why it made them.
Yesterday, I simply cut & paste the entire PlexMedia folder over to PlexBackup via the Cinnamon GUI to make an identical backup. Then I planned to make a script to run in cron each night to backup changes. I don’t know what is different (ownership/permissions/?) but when I saw RSYNC copying literally everything, I stopped it with CTRL-C and every time I rerun it, it picks back up where it left off lol. I tried a different set of switches then and it said skipping directory PlexMedia, so I restarted:
and I guess I’ll let the damn thing re-copy all 10TB again. For the life of me, I cannot understand why there are 100’s of ways to do the same thing in Linux. I get that it’s flexible. I also get that there are many ways to accomplish the same task in ANY OS. But Linux seems to get crazy-stupid with how many ways you can do the same thing. This is coming from a long time Windows user and DOS user before that.
EDIT: Since this was coming from a Windows system, both file systems are NTFS, if that makes a difference.
You’re not using the -c/--checksum flag, so rsync is definitely using file size and last modification time to determine which files to transfer. But you don’t have to guess what the difference is though, if you tell rsync to print out a log, and add the -n/--dry-run flag like @WorBlux suggested, rsync will tell you, per file, why it thinks source and destination files are different.
It isn’t that bad as long as each commit is tiny. I’d rather see tiny commits than one giant commit if the commit messages aren’t good. Of course, one giant commit with actually readable commit message would be better but if wishes were fishes…
I tried using the log command with -ni earlier but it wouldn’t run. Turns out, I’d pressed CTRL-Z and not CTRL-C and RSYNC was paused or something, so when I tried the new command it failed.
I then used:
rsync -rtuv --progress --delete-during /disks/PlexMedia/Media /disks/PlexBackup/Media
and it began copying everything over. Out of curiousity, I looked at the directory structure and it was creating the /Media folder and its subfolders inside the other /Media folder(/disks/PlexBackup/Media/Media). I deleted that, added a / to the end of my source and destination (I think) and it began deleting files from the destination.
I appreciate the help you guys have given me, but I’m going to just stop where I’m at and pick back up with a fresh look tomorrow. I’m just too frustrated right now to waste both our time.
Don’t get frustrated Tabun, keep at it you will eventually learn RSYNC and how to work around all its little quirks. while it sounds like I have more experience using Linux than you, I am also a beginner, There has been plenty of situations where I to have been tempted to just give up on Linux altogether and just go ahead and install Windows 10 on my desktop. But I am determined to learn Linux and all its quirks, so I won’t install Windows 10. The rewards are well worth the effort. Also maybe Magnus_Python will post a gif of Captin Picard when you finally able to get RSYNC to do what you want it to do.
I have a suggestion, Tabun may want to create two virtual machines and just practice using RSYNC until RSYNC does what he wants. After he masters that then do ahead and do it for real.
Yup. I actually had RCLONE working but it seemed a bit slow, so I tried RSYNC. (RCLONE is built on RSYNC but is a bit more user friendly) I’ll hit it again tomorrow. The funny thing is, this is the part that I figured would take a couple of minutes to set up but
Everything is good now. I waited a couple hours and then went back to RCLONE and it took an hour to replace what I accidentally deleted with RSYNC. Now, when the cron job runs with no files having been added to the source, it finishes in a second or two, as I thought it should. Further tests after adding files/directories to the source were immediately detected when I ran my shell script and copied to the destination.
I already had Linux Mint on the mini PC I’m running Plex on, so now that I’ve ironed out the kinks, I’ll probably switch to Ubuntu Server and set it up headless.
I wanted to thank you personally for mentioning RCLONE. RSYNC could probably have done exactly what I needed (if it wasn’t for the ID-10-T errors I made) but RCLONE is practically a drop-in replacement for ROBOCOPY.
It sounds like your close??? Walk back your steps?
