How to maintain the integrity of cold storage HDD data?

Long time L1 youtube supporter and I created an account here to ask this question:

How do I mitigate bit rot or other threats to the data of my offline hard drive backups? I am a noob compared to the folks in this community and so please consider that in your response. As is I make periodic backups of my data to various HDDs some of which are then powered down and stored on a shelf. I am concerned that over time the data on these drives may loose its integrity.

Thank you

This is as far as I know, and I’m not an expert by any means. I store my hdds in anti-static containers or anti-static bags, in a dry environment, at room temperature and not near any electrical or otherwise magnetic material.

1 Like

Thanks for the reply.

However my question is more about the integrity of the data over time and not so much about the physical care of the HDD.

Oh, so you mean when you do a backup you want to be sure it is not messed up in the transferring phase? You could hash it and compare, even though it’s probably going to quite some time.

To reiterate: If you keep your cold drive safe, for instance following the points I set up, I first of all don’t think anything will happen to your data, and second, I don’t think there is much beyond that you can do.

Edit II: To my knowledge, bits don’t randomly flip when the drive is cold. Okay, a stray something from a solar wind can flip a bit, but that is only mitigated by storing multiple versions of your data, preferably on different kinds of storage.

turn them on at least once a year and preferably in zfs environment. with redundancy through gluster or unraid.

4 Likes

Either have your backups in a box that supports zfs (so basically a NAS), or have multiple copies which are zipped and make checksums of the .zip files. That way you can verify that the .zip is intact when you unpack it again later.

I bought the paid version of MD5 and SHA Checksum Utility and use it to make checksums of my movies and series’ backups.

turn them on at least once a year and preferably in zfs environment. with redundancy through gluster or unraid.

Thanks for this. Unfortunately most of that is over my head. ZFS, ok ya that is a file system. Gluster? I do not know and will have to google. Unraid… something Linus does. :slight_smile: I have a lot to learn and there does not appear to be a “simple” solution to protecting data over the long haul. :frowning:

Either have your backups in a box that supports zfs (so basically a NAS), or have multiple copies which are zipped and make checksums of the .zip files. That way you can verify that the .zip is intact when you unpack it again later.
I bought the paid version of MD5 and SHA Checksum Utility and use it to make checksums of my movies and series’ backups.

Thanks for that I will consider what you suggest. A NAS with ZFS sounds like a plan but it also needs the complimented with cold storage IMO.

TBH I am a little leary of zipping data. I zipped my LTC passphrase and then the zip corrupted… ironic considering my thread I know. Anyway, I cannot access my wallet and there a 60 coins in there. :disappointed_relieved:

If I were concerned about data retention (which I am not, everything is temporary, consider it gone) I would probably use ZFS on a single drive, and make multiple copies of the file on that single drive. Then copy that drive once or twice.

You cant prevent deterioration or accidents. Having multiple copies is the only way to keep something for an extended period of time.

You could also encrypt the most crucial data and keep copies on public platforms like dropbox and let them deal with the data integrity…

Edit: Other file systems may use copy on write, but I know some will make a reference to the same data. Thats why I recommended ZFS.

On top of what everyone has already mentioned, it also depends on how long you plan to keep everything. If you’re planning on under 5 years, yes, plugging it in and having a few drives for redundancy would work. However, drives do get old, even when they’re sitting on a shelf. If you need enterprise level storage archival, you might want to think about a tape backup. Tape drives can be found pretty easily, as many people try to upgrade when they can, so the older tech gets dropped for fractions of the new stuff.
A brand new tape drive might be $1-4k for an LTO7 drive, and the LTO7 tapes that hold 6TB uncompressed data or 15TB compressed data are $80.
An older LTO2 drive, even in good condition, could be found for around $50, and the LTO2 tapes that hold 200GB uncompressed data or 400GB compressed data are $12.

The benefit here is that tapes are made to last 15 years, easily, rather than the MAYBE 7 years a hard drive could. This is the solution that many companies use when they need to satisfy something like HIPAA to keep data for lengthy amounts of time without it being in use. Essentially this solution is plug and play. Depending on the drive, you may also need a pci card and/or drivers. Let us know if you choose to take this route, and we can maybe help guide you to a solution that fits your needs.

Just a thought, since you didn’t really clarify how long you needed to keep your data.

Just a thought, since you didn’t really clarify how long you needed to keep your data.

As long as possible… my lifetime for sure. I want to preserve my photos and videos as well as other docs indefinitely within the best practices available to me. My main issue is my ignorance as I am no computer genius.

I would like to ask about SpinRite, does it do a good job of refreshing the surface of a platter? Can it mitigate bitrot?

Honestly, nothing will be safe for a lifetime. You’re either going to have to run a ZFS or other RAID and keep swapping out drives as they go bad, or swap out tapes. I would recommend the RAID/ZFS option if you’re doing pictures. Physical tapes can be a pain sometimes for small data restores.

Anyone have any input about M-disc (here’s a review), will theoretically last 1000+ years. You need to have an M-disc writer, but most DVD-players should be able to read them… The writer can be had for $130 (2sec google search, you mileage may vary), the discs are $5 for 25GB according to the review. I can find 4x50GB on amazon for $70.

1 Like

@Superuser1 like you, I want to archive my data for the remainder of my lifetime. Given health issues I’d estimate I’ve got about 10 years or as an extreme outlier, another 30 years (which is really an extreme estimate)

In any case, the approach I’m taking is the following:

  • Primary ZFS store // this is currently in operation as a FreeNAS build, Raidz2 with 8x WD Red 4TB NAS drives; these drives are only about a month-new.
  • Secondary FreeNAS replication server // This will be in 2018. ZFS > ZFS replication. Minimal effort to setup.
  • Primary backup (rsync) Synology DSM1815+ // This used to be my secondary Synology backup (from the 1515+ primary), but I’m making it the primary backup as RAID6 with 6x WD Red Pro 4TB drives.
  • Manual (rsync) backup of ZFS mount-points to WD Black drives (offline storage).

This leaves another 5x WD Red Pro 4TB drives from the 1515+ to move into the 2018 replication server. These drives are 2-years, 9 months old though.

I also have some Samsung T3 SSDs, but these are just temporary data ‘mules’, for moving ingest data around. I’ll probably have to replace the WD Blacks in about 4-5 years.

Plus replace the WD drives as errors start popping up.

Well for starters you run the fuck away from ntfs. If you’re scared of bitrot you should be scared of it as well. Secondly, you don’t. I know it sounds dumb but 20 years down the line you’ll have a demagnetized hdd and it’ll all just fall off.

Heres what I do. Take your most important files. THE MOST. Throw them on mega.nz and forget about it. If you need them, there they are. Next, if you have music up the wazoo, bounce it between devices and sync back your phone to your pc every year. If theres anything missing i know itunes goes ‘oh hey look at that’ and fixes it. But even then just copying the folder around. And if its reallywthat important to you, go buy some M disks, burn that shit on them, put them in a safe.

Tell that to the people who stored their data on MegaUpload. Once the feds raided that place, they simply shut down the servers and everyone lost access to whatever they had on there.

Well if you don’t know the legal structure they set up there or that half the people there are from monova and the pirate bay because they wanted out of the sketchy bs, then apply hat logic to google drive.

Fine then. Op put your shit on icloud in an encrypted tar. I can trust apple with data.

Well this isn’t a free solution, but you could also setup an AWS account.

One way, would be to plonk your data on S3, but I generally feel costs are pretty high and there’s guarantee against bit-rot there either.

So you’re better off setting up an EC2 instance with encrypted block-storage attached. You would want to bootstrap the EC2 instance either via a shell-script or say Ansible, so that you can replicate this on a fresh instance should the virtualised resources that represent your EC2 instance ever experience ‘issues’.

You’d also want to script (via the AWS SDK) a snapshot-strategy of the block-storage volumes. ZFS or cryptsetup are a couple options. YMMV regarding cost though.

I have also been looking for a solution to bit rot for backups. In my search I have come to the conclusion that as good as modern HDD are, there is no solution to this problem if the HDDs are sitting in storage somewhere, even if environmental conditions are perfect.

As others have said, tape and m-disc are better mediums for long term storage, but even these would need to be replaced on a regular basis. My current active solution is putting all my important data on an ReFS volume on my desktop using Windows Storage Spaces. If this wasn’t my main PC that I needed to use for other things, I would have gone down the ZFS/BtrFS route as a dedicated NAS. Currently doing backups to single 6TB drives, one onsite and one at a trusted location just for my 3-2-1 rule. Test the backups on a monthly basis to make sure they didn’t get corrupted and it seems to be doing ok for my needs now.

I would say there is no “free” way to protect against bit rot unless you are rewriting all data periodically to your backup drives. I can say a cheap way of protecting all your data would be some sort of cloud storage solution. If you really don’t need to touch the data unless it’s an emergency, you could go with Amazon Glacier. Current rates in the US are $.004/GB/month for storage and they charge somewhat less than that for retrieval. For my money, I would get Backblaze and be done with it. Unlimited storage for a flat monthly fee of $5 per computer. Pay more in advance and I think you can get 2 years for $95. If you want to get fancy with it, you can even set your own encryption key so even backblaze can’t see or access your data.

If you don’t like the cloud and you like ease of use, synology units that support SHR (Synology Hybrid RAID, marketing speak for BtrFS) are easy to use and allow dissimilar drive sizes to be used. You can also replace these drives with larger ones as your needs grow. Can even set up one in your house and one at a trusted location and have them sync over the internet. SHR will actively protect against bit rot with checksums of metadata as well as the data itself. The hardware isn’t cheap, but it’s stable, easy to use, and well supported. And you can use the drives you have laying around if you really want to.

Unfortunately bit rot really comes down to physics and there is no cold solution that will be able to protect against it.