Large Storage Help (1-2-3 question)

So I’ve had a 12TB drive for a while, filled it up with movies, so I bought a 26TB drive. Reading up on RAID (yes, I know I need more 26TB drives to do it properly instead of wasting 14TB of space) I found this post and there’s a comment that says RAID is not a backup, to still have proper backups just in case.

So my questions is: how do I backup 38 TB worth of data in an economical way? :joy: (don’t have many friends into data hoarding as I am sooo can’t really ask trade space with them. Anybody have recommendations?

First, only back up what you need. Any copies of readily commercially available movies are pretty much stupid to preserve. Private movies of, say, your wedding day? Unless you want to trigger a divorce you better back that one up. :slight_smile:

Second, you softraid with Z1.

Third, well… Buy magnetic tape and/or optical storage. A stack of Blurays with each storing 50GB is still better than nothing, though you will need about 800 discs.

Fourth, wait five years for 100 TB SSDs to become affordable enough?

Other than that… GLHF.

1 Like

I’d take those reddit comments on raid with a mountain of salt, there are a lot of false assumptions and bad zeitgeist they are propagating there. The only thing I’ll agree with them on is raid is not a backup.
The more finicky your raid setup is the more you’ll need those backups, and there is a large gamut of reliability in different raid setups.

Alot of the homelabber and datahoarder types will use their previously decommissioned server as the backup to their newest server… but this won’t help you if you are just starting.

The cheapest IMHO minimum viable way to do this would be to copy over the 38TBs worth of data to a couple of loose HDDs and then store them properly.

2 Likes

Correct.

We were all here at some point, first time expanding our storage needs and realizing, “I don’t want to loose this stuff.”

RAID is NOT a backup. Without a long winded spill, it boils down to when (not “if”) your data gets compromised (ransonware’d, virus, lightning storm zap, etc), RAID means it’s all corrupted together. What happens in one place corrupts the other replicated data instantly. Poof, gone. Or you just loose the array with two drives falling. Lots of things can easily go wrong with raid sets, especially if built on the cheap.

My advice is to read up on the “3-2-1 backup strategy.” Skip 10 years of pain of manual USB HDD copying for somewhat of a backup strategy and go straight to the 3-2-1 mindset.

Start by separating the data into importance. User data (friends, family, phone photos backup for 20 years, etc) goes into /users/ . Home 8mm movie digitizing goes into /users/home-movies, and Linux ISOs goes into /media, work goes into /work and so on.

Now, you know the rough sizes of the most important to least important.

Start with a second copy somewhere locally. This can be an old laptop you install Linux on, or as simple as an “temporary” external drive. “Temp” because you don’t want to leave it connected - if you get compromised, ransonware’s, or a bad electrical lightning storm zaps your electronics, you don’t want your only backup connected.

What happens if a house fire burns everything? Got your “1” offsite copy available?

Can you start to see a pattern here with compromised data storage? Always assume it’s going to fail, every level of the “3” in 3-2-1, and plan around it failing. So if you loose your backup laptop, or the offsite data went offline, etc, you’ll have other copies to continue on.

In my 3-2-1 strategy, I evaluate 3-2-1 for each backup type. Let’s take my /users…

  • /users goes to into an ZFS mirror pool (again, raid/mirror is not a backup!) as one of the 2 storage mediums.
  • The second onsite storage medium (the “2” in 3-2-1) is my dedicated backup server. On a budget, this can be an old laptop you’ve formatted and encrypted the HDD on as user data tends to me on a small side.
  • And for the “1” in 3-2-1, I have a couple of external drives I rotate every few weeks/month to a friend’s house. (And I do the same for their off-site backup drives)

Since the data is all encrypted, the offsite storage is safe.

Now, my 92 TB of Linux ISOs has a totally different 3-2-1 strategy. In short, I can’t do the “1” as it’s just too costly. I previously started an Amazon AWS Glacier archive but that quickly slowed down and ran up a bill.

It sucks to duplicate the cost, but it’s the only way. Since it’s so expensive, I had to forego the “1” strategy which says i’m ok losing all that data in a house fire. Not so for my users and home movies and kid’s drawings!

My scripts actually wake up the backup server on a schedule for weekly backup and then shuts it down.

3 Likes