Homeserver offsite photo backup and x11VNC fix needed

Software & Operating Systems Linux
1

Hi,

i’m running my homeserver on an old workstation with debian 9 with all of my files on a RAID1 equivalent ZFS pool (2x 4TB WD Red)

I’m no longer sure if that was even a good idea after reading these three threads [ 1 ] [ 2 ] [ 3 ]

I would like to have a redundant Hardware setup even though it’s not really necessary

Anyway it is mainly used as Mediacenter PC, Nextcloud server and NAS

And currently I have two problems:

The internet connection breaks:
I’m able to fix this by adding nameserver 1.1.1.1 to /etc/resolv.conf
but after every restart it’s gone as if something resets the config to blank

to controll the mediaserver i use Kore (android app), my Xbox360 controller (through Steams Desktop Controller setup) and sometimes VNC with an VNC viewer app on my phonesometimes when I don’t disconnect and leave it connected I’ll get disconnected without being able to reconnect due to "The connection closed unexpected"I have to SSH into the machine an restart x11vnc with x11vnc -R stop to regain connection

Furthermore I would like to setup Offsite backup for photos

To save space and not requiring to purchase extra space I would like to compress and encrypt it for privacy reasons

So I would need to monitor my Nextcloud autouploads aswell as a manual folder on the NAS part

I have two Users on Nextcloud

I would need a script that monitors the three folders for new uploads with a 24h cycle

Copy the new files to an temp folder where they get compressed and encrypted and after that upload it to a Dropbox or Google Drive account and than delete the compressed files after upload so only the original file is on the harddrive

I found tow guides that could be useful one is for the notify and the other is on image compression

I’m a somewhat between beginner and intermediate Linux user, so a Step-by-Step would be welcomed but I’m still somewhat capable with no experience in scripting

2

Some distros have an /etc/resolvconf.conf or something similar to generate your /etc/resolv.conf. Idk about Debian, but if it doesn’t you can edit your /etc/network/interfaces according to Stack Overflow

3

Are you using RAIDZ-1 (raid5) or mirror (raid1)? Those are two very different things.

You should be using mirror.

Sounds like a network manager problem. I don’t know what your network management software is, since I don’t know much about your setup. Are you using a gui? If so, use network manager (nmcli or nmtui are for SSH sessions) to configure the name servers.

I seem to recall @oO.o using something for something like this. Care to chime in?

You could use deja-dup as well.

4

rclone is what you are looking for as far as copying to Google Drive and encrypting (they might have compression options as well). It works like rsync.

Dupicati is another option. It’s structured differently. Up to you which is more suitable. Here is a thread I wrote on getting Duplicati running on a headless CentOS server. It won’t be exactly the same on Debian, but should be close.

As far as temp folder and deleting goes, you’ll probably need to script that yourself.

1 Like
5

There’s a variety of fixes for this, but probably the easiest is to simply add the relevant DNS servers to /etc/network/interfaces alongside the interface configuration :-

auto enp0s31f6
iface enp0s31f6 inet static
    address x.y.z.2
    netmask 255.255.255.0
    gateway x.y.z.1
    dns-nameserver 1.1.1.1
    dns-nameserver 8.8.8.8
    dns-search ${list-of-domains}

The dns search list is optional.

For compression, I’d start by making sure ZFS compression is turned on for the filesystems that you’ve created :-

✓ msm@scrofula» sudo zfs list | grep osimages
p0/osimages                 35.8G   710G  35.8G  /p0/osimages
✓ msm@scrofula» sudo zfs set compress=on p0/osimages
6

Thanks for the feedback, I added an DNS server in the GUI networkmanager which solved the DNS error

i’ll look into rclone, hope I can get it working like I want to :grin:

thanks

7

A suggestion for your offsite backup.

I have an email & Nextcloud server running on a Udoo x86 (think x86 single board PC like a Pi on steroids) with the data stored on a FreeNAS. My offsite backups are in an AWS S3 bucket.

This costs $0.024 per GB a month. You can do it cheaper with different options.

All I did was install the AWS CLI on the Udoo x86

https://docs.aws.amazon.com/cli/latest/userguide/installing.html

And write two scripts to upload / update / delete any changes each night using cron.

This one backs up the email and nextcloud directories with their DBs and runs under the root account

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
# Backup postfix, compress and copy
# Backup postgrey, compress and copy
# Backup dovecot, compress and copy
# Backup email, compress and copy
# Backup nextcloud, compress and copy
# Backup DBs, compress and copy
# Remove backup folders
# Sync all compressed archives to AWS S3
# Remove backup files

mkdir -p /tmp/backup/postfix_backup
rsync -r /etc/postfix/ /tmp/backup/postfix_backup
tar -czf /tmp/backup/postfix.tar.gz -C / tmp/backup/postfix_backup
rsync /tmp/backup/postfix.tar.gz /mnt/FreeNAS/Backups/SERVERNAME/postfix.tar.gz

mkdir -p /tmp/backup/postgrey_backup
rsync -r /etc/postgrey/ /tmp/backup/postgrey_backup
tar -czf /tmp/backup/postgrey.tar.gz -C / tmp/backup/postgrey_backup
rsync /tmp/backup/postgrey.tar.gz /mnt/FreeNAS/Backups/SERVERNAME

mkdir -p /tmp/backup/dovecot_backup
rsync -r /etc/dovecot/ /tmp/backup/dovecot_backup
tar -czf /tmp/backup/dovecot.tar.gz -C / tmp/backup/dovecot_backup
rsync /tmp/backup/dovecot.tar.gz /mnt/FreeNAS/Backups/SERVERNAME

mkdir -p /tmp/backup/email_backup
rsync -r /var/mail/ /tmp/backup/email_backup
tar -czf /tmp/backup/email.tar.gz -C / tmp/backup/email_backup
rsync /tmp/backup/email.tar.gz /mnt/FreeNAS/Backups/SERVERNAME

mkdir -p /tmp/backup/nextcloud_backup
rsync -r /var/www/html/nextcloud/ /tmp/backup/nextcloud_backup
tar -czf /tmp/backup/www_nextcloud.tar.gz -C / tmp/backup/nextcloud_backup
rsync /tmp/backup/www_nextcloud.tar.gz /mnt/FreeNAS/Backups/SERVERNAME

MAIL_DB=mail
NEXTCLOUD_DB=nextcloud
USER=root
PASS=YOURPASSWORD

mkdir -p /tmp/backup/SQL_backup/sub

mysqldump --opt --user=${USER} --password=${PASS} $MAIL_DB > /tmp/backup/SQL_backup/sub/mail.sql
tar -czf /tmp/backup/${MAIL_DB}.tar.gz -C / tmp/backup/SQL_backup/sub/mail.sql
rsync /tmp/backup/${MAIL_DB}.tar.gz /mnt/FreeNAS/Backups/SERVERNAME/

mysqldump --opt --user=${USER} --password=${PASS} $NEXTCLOUD_DB > /tmp/backup/SQL_backup/sub/nextcloud.sql
tar -czf /tmp/backup/${NEXTCLOUD_DB}.tar.gz -C / tmp/backup/SQL_backup/sub/nextcloud.sql
rsync /tmp/backup/${NEXTCLOUD_DB}.tar.gz /mnt/FreeNAS/Backups/SERVERNAME/

rm -rf /tmp/backup/postfix_backup
rm -rf /tmp/backup/postgrey_backup
rm -rf /tmp/backup/dovecot_backup
rm -rf /tmp/backup/email_backup
rm -rf /tmp/backup/nextcloud_backup
rm -rf /tmp/backup/SQL_backup

aws s3 sync /tmp/backup s3://s3bucket-001.manymachinesonix.net/YOURNAME/FreeNAS/Backups --delete

sleep 10

rm -rf /tmp/backup

and this one does the same for the actual data on the FreeNAS which has to run under www-data

#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
aws s3 sync /mnt/FreeNAS/nextcloud/data/[email protected]/files/bin s3://s3bucket-001.yourdomain.net/your-name/bin --delete
aws s3 sync /mnt/FreeNAS/nextcloud/data/[email protected]/files/Desktop s3://s3bucket-001.yourdomain.net/your-name/Desktop --delete
aws s3 sync /mnt/FreeNAS/nextcloud/data/[email protected]/files/Documents s3://s3bucket-001.yourdomain.net/your-name/Documents --delete
aws s3 sync /mnt/FreeNAS/nextcloud/data/[email protected]/files/Notes s3://s3bucket-001.yourdomain.net/your-name/Notes --delete
aws s3 sync "/mnt/FreeNAS/nextcloud/data/[email protected]/files/Pictures/Camera upload" "s3://s3bucket-001.yourdomain.net/your-name/Pictures/Camera upload" --delete
aws s3 sync /mnt/FreeNAS/nextcloud/data/[email protected]/files/Pictures/Misc s3://s3bucket-001.yourdomain.net/your-name/Pictures/Misc --delete