Ultra Tiny Offsite ZFS Replication Target: Zimaboard 2, Tailscale, Sanoid & Syncoid

This thing supports ZFS!

image

My mechanical/sata disks were sda and sdb so I used this command:

zpool create \
  -o ashift=12 \
  -O atime=off \
  -O compression=lz4 \
  -O xattr=sa \
  -O logbias=throughput \
  tank \
  mirror /dev/sda /dev/sdb

You should understand the implications of this configuration. xattr=sa stores the xattrs in the dnode, which is probably better for lots of small files.

If you want to be extra correct long-term, use by-id instead of /dev/sda//dev/sdb, but generally on reboot ZFS will use these device names anyway.

s -l /dev/disk/by-id/
# then something like:
/dev/disk/by-id/ata-ST8000...   -> /dev/sda
/dev/disk/by-id/ata-ST8000...   -> /dev/sdb

… can use that for device names instead of sda and sdb.

As this is a a backup appliance, if you prefer zero redundancay (CAUTION)

zpool create -f \
  -o ashift=12 \
  -O atime=off \
  -O compression=lz4 \
  -O xattr=sa \
  -O logbias=throughput \
  tank \
  /dev/sda /dev/sdb

As this is a replica I also don’t mind sync being disabled globally.

zfs set sync=disabled replica

Sanoid & Syncoid installation is a bit more annoying. apt isn’t available, really,

and that’s the reasoning.

Sanoid and Syncoid on ZimaOS/CasaOS

It is possible to install and configure sanoid and syncoid on CasaOS/ZimaOS. However for our stated use case of using a Zimaboard2 as a remote backup replica, with tailscale, it isn’t strictly necessary.

If you do want to do that, especially just sanoid for zfs snapshots and replica management (i.e. NAS usecase) then I would recommend setting it up in /DATA. /DATA/bin is already in the system path and /etc/ is not part of the read-only filesystem. So it can be done, without too many modifications.

sudo git clone https://github.com/jimsalterjrs/sanoid.git /DATA/sanoid
cd /DATA/sanoid
mkdir /etc/sanoid
mkdir /DATA/bin
cp sanoid.defaults.conf /etc/sanoid/
touch /etc/sanoid/sanoid.conf 

Documentation:

You would, ideally, set it up on the system you want to push replicas FROM.

Typical Setup

Source (where you run syncoid):

  • syncoid installed
  • SSH key set up to target
  • ability to run zfs send

Target (remote replication host):

  • Just ZFS
  • A receiving pool (e.g., tank)
  • SSH access (key-based strongly preferred)

With Tailscale on both the Source and Target you can ssh directly from the source to the target, using the tailscale IP address.

I do not recommend setting up the Source to be able to pull from the Target via SSH because of the security implications. It is possible to configure SSH to only allow the sync command, but that’s outside the scope of this how-to.

For example, this should work from the Source system:

syncoid -r -c tank/data root@zimaboard2-host:tank/data

…and will not require Sanoid or Syncoid on the zimaboard2 since it uses built-in zfs send functionality.

Summary

Host Needs Syncoid? Needs ZFS?
Source (push mode) :white_check_mark: Yes :white_check_mark: Yes
Target (push mode) :cross_mark: No :white_check_mark: Yes
Source (pull mode) :cross_mark: No :white_check_mark: Yes
Target (pull mode) :white_check_mark: Yes :white_check_mark: Yes

Note: The overhead of ssh is significant. For faster connections, it is possible to setup direct connections such as with mbuffer and zfs send. There are some old writeups from me on how to do that on the forum, and I can revisit writeups on doing this if there is enough interest or it is unclear on what the steps would be.

5 Likes

I have been planning for something like this as an alternative to Storj/Backblaze backup subscription but then in my local backup (with an Odroid H4+) I had some issues with some errors, presumably with having too much uptime without using proper ECC. I have scheduled for it to do automatic reboot from time to time.

I wish for something like this, but with ECC (even if it is DDR4) and cheap as well. I haven’t had the time to scour the internet for good enough and cheap NAS SBCs for this to come to fruition. At the price point I am targetting, it would be AMD Ryzen-based but I have not found a good one yet.

the aliexpress ryzen pro ddr5 sodimm platforms are the only ones I know of that have the best shot at support. otherwise its prettmuch asrock deskmini (last gen) or this gen for confirmed ddr4ecc/ddr5ecc sodimm support.
the bigger deskmeets with 4 full-size dimm slots could be just the thing but I never managed to get my hands on the am5 version.

1 Like

When I setup my ZFS offsite backups with Syncoid I opted for a pull configuration so that if the source server is compromised the attacker can’t just use the SSH key to hop over to the server with the backups and trash them.

With this push setup you probably shouldn’t be connecting to root on the backup server. Make a separate user and use zfs allow to give it the permissions it needs. This guide was really helpful when I was setting this all up.

Yes this, but also, ssh authorized keys can have only allowed commands, too, which really locks things down even more.

You can go a step further and force a command when an ssh key is used. And then one more. That command can be a shell script that will examine the command you meant to run, sanitize it, disallow it, redirect paths etc. I use a system like this to allow certain keys to for example, run one or two scripts or push files, but only to certain paths and only certain types. It may seem like paranoid overkill, but it certainly helps me sleep at night.

This is my Zimaboard 2 build, I had it paired with an IBM M1210 (aka LSI 9300-4i) which is the only PCIe 3.0 x4 HBA out there that I could find and it has only costed U$19 (shipped!). As of right now I have about 300Tb of data connected through it and it is working quite stable without any issues at all.

You might wanna snag one of their 120W DC power supply too if any of you are interested in it as I managed to power 4x SAS enterprise SSD’s (EMC 7680) without any problems and it ran very stable during RAID calculations. So I’m pretty sure that @wendell could have had better luck with other PCIe cards had he been using the 120W PSU instead.

Oh and let’s not forget that the possibilities are pretty much endless if you’re into 3D printing too. Anyways, great review and thanks for going through the details.