For every TB i’m excited about adding to my main NAS (will be around 60TB), I need an equivalent stored somewhere safe… Something I didn’t consider for far longer than i’d like to admit…
I have bad internet connectivity limited space in the apartment. I have Cat6 ethernet in the walls, the only viable options: a wardrobe or next to the main NAS in the lounge room.
Way I see it, I have 3 options:
Build yet another ZFS TrueNas system with capacity to take on the same data, with intelligent VDEVs / Z2 etc. and have it on the LAN. Cons: not offsite, expensive
Option 1 but have it offsite and connected via VPN. Cons: Need better internet, need someone savvy to host it, expensive
Periodically (weekly/monthly) replicate the ZPool onto externally connected media. I like this option the most, but to cater for the size of the ZPool i’d need a device to plug into the NAS with enough space (60TB+) and that can handle ZFS replication. When the device is plugged in, each drive can be seen reliably and a pool created from it. Export the pool, disconnect and relocate offsite.
What type of hard drive enclosures work: non-USB, more like eSATA or SAS maybe?
Thinking this is the cheaper option, but I might be just far too biased and too used to: backup = plug in something in, rather than backup = be more intelligent.
Does anyone do option 3? Any tips on devices for this?
Oh going cloud based would be way less hassle that’s for sure - really wish it was more affordable. I got excited last night seeing Backblaze Unlimited, but realised I’d be connecting a NAS not a home desktop, bummer! I’ll check if there is a server rental setup that could be affordable, thanks for the tip!
B2 isnt terribly expensive, but 60TB, it adds up quickly. Is your data compressible? If so, restic might offer you good results, but thats just software, not backup destination.
I would say partially. Data types will be mainly RAW images (.NEF .ARW .CR3) processed images (.jpg), RAW 4K video, transcoded media, 3D point clouds (these have good compression ratios), ISO snapshots for computer backups
You could try to build a reasonably small machine with 22TB drives in it, running TrueNAS, that you connect to your home and move about.
But I guess you’ll be moving the bulk of the storage only once and update it so if your home and the offsite location have both 1GBit symmetric is not gonna be that big of an issue keeping the data synced.
If you have no internet where your offsite backup is located I’d stick to move that machine around. An ITX board and a flex PSU ain’t gonna make a machine much bigger than a JBOD enclosure anyway.
There are also services that allow you to deploy your own machine at their site. High maintenance but, hopefully, lower cost since you’d be paying for space, electricity, internet and cooling. Can’t recall who offers this option, sorry for the incomplete information.
Thanks for these ideas! I suppose this secondary NAS/DAS wouldn’t need to be high spec beast - so could well be a lot cheaper than I first thought! Good idea having high capacity drives to save on drive bays + power.
I found this really interesting too, 3D printed brackets to outfit an old PC case! Pretty awesome. Would just need that External SAS connector, power supply and some fans.
For an off-the-shelf SAS JBOD, I also saw these from iStar, could be a winner!
RE: internet, such a pain, i’m in good old Australia with its non-symmetric fibre… 1000/40mbps Could get a business tier fibre, but its pretty $$$
How much data do you expect to write per day? If it is not to very much you could do the initial replication on the same LAN and then move it to an external site. I kept a personal NAS with nextcloud/some other services in sync with a remote system over a 1/10Mbit line, and we where 2 families at that place. It sucked, but it was possible
I am not so sure about the tech savyness needed either. Once configured to receive snapshots it is somewhat self-running, and you could always do admin of it over VPN/SSH. I am remote administrating my Dads NAS since 4 years. Works pretty OK. Do you strictly need the VPN? Key based SSH is pretty secure.
But yeah, it is not going to be cheap with that amount of storage. Taken a look at second hand HDDs? You can buy them for quite a lot cheaper and just add a bit more redundancy to compensate. The remote site should have 1 or 2 hot spares to use in case of disk failure.
If you win the lottery, you can think about 40x Micron i400 and copy your data to these cards. 40 pieces is still quite compact in terms of physical size and therefore easy to relocate.
More seriously… to tell the truth, there is currently no good and cheap option for storing large amounts of data remotely IMHO.
Cloud… yes, but with the number of TB the price starts to rise, and you always have to have a fast down/up connection so that it ultimately has a reasonable operation time.
A second machine with many disks kept in a different location, regardless of whether online or offline, is again a significant cost and the risk of failures and the need to maintain the equipment.
Tapes… They are also not as cheap or as capacious as you would like. But for large amounts of data for cold backup and storage in another location, there is probably nothing better except for cloud solutions.
Solutions based on DVD/BR or SD card are also becoming expensive and not very convenient with such amounts of TB.
The easiest thing would be to simply buy the right amount of HDDs for your needs and copy the data onto them in batches and move the disks to another location.
The whole thing gets a bit more complicated if it’s supposed to be a bare mirror copy of zfs and not some dedicated backup copy.
ZFS replication is great. Once the initial replication is complete, subsequent replication tasks will only send the delta. The replication (zfs send) is totally automatic, so your upload speed is honestly not that important.
What is your typical upload speed? How much data do you expect to add/modify on a daily or weekly basis?
Anecdotal, but at my former apartment, my upload speed was not great. Often about 10-15 Mbps. If I added a ton of data to my NAS, it would often take several hours or even a day or two to replicate that new data. That said- who cares how long it takes so long as it generally works?
cannot elaborate, but that’s a KILLER deal when used appropriately
We have clients’ workstations housing 2TB local being backed up for $8 / month
Takes some genuine engineering to not break apps and shit working with the BackBlaze exclusion policies, but I’ve done the math and cannot host @ $10/TB per month with 5 9’s uptime, much less half that.
I may start hosting to feed niches (local / encrypted / colocation) as I need to deploy my servers in 6 locations across the U.S. for my next-next trick.
Easily
I pay $10 / TB of RAW storage (22 TB drives)
that’s no compute
that’s no networking
that’s no availability (double the above summation calculation)
Average storage time is 3 years for typical cloud, TRIPLE that for offsite archive.
So I have to do the above 3 times to maintain minimum archival storage.
Now, there’s economies of scale at work and creative engineering (deduplication of files client versus server side) without consequence.
I cannot move everything to cold store and still offer the same service level.
Ingestion has to be NVME backed, and storage can be spinning discs. But a minimum availability precludes sectioning.
And as soon as large client side encrypted files (in-compressible by most algo’s server side) enter the mix, shit hits the fan.
Long story short, it’s approximately 9x more expensive to START a cloud based storage company (fully featured) than on-prem.
This is a fair point. I’ve made some assumptions about OP’s offsite location based on my own setup.
I have my offsite TrueNAS at my family’s house, its about an hours drive away. In this scenario, it is pretty easy to drive over, bring the NAS back and then restore via LAN.
Very true. But as long as you do not have a gigabit connection driving somewhere and restore is going to look non-stupid from a time perspective. I interpret “bad internet” as under 100mbits, which is going to take about two months for a full restore (and initial upload). Even at gigabit you are looking at close to a week, and likely a polite letter from your ISP asking if you really should be on a non commercial connection. (Oh wow 60TB is a LOT of data, I guess schedule a long stay at your friend/parents house Or invest in a 10Gbit NIC ).
The main point for both of our posts is that a bad internet connection does not make “dump a box at the parents/inlaws/friends house” impossible. The only thing needed is that the connection can keep up with the data growth. A 10/50 connection at both places is going to keep up with 100GB/week, even if scheduled to run nights only
Personally, I see that approach or cloud storage to be the only “real” solutions. A box in the same apartment is close to a waste of money from a data protection perspective and having a “box of disks” that is stored somewhere outside the apartment and brought in for updates is inevitably going to be
Planned to be updated weekly
Actually done monthly the first year
Some day OP is going to be wondering if the last copy was made this Christmas or last Christmas
Australia with its non-symmetric fibre… 1000/40mbps 
Could get a business tier fibre, but its pretty $$$
Or invest in a 10Gbit NIC ).