I have a large collection of images around 8TB worth. I have scanned photographs dating back 40 years. I want to protect them from bit rot. So was thinking about using ReFS because of its corruption healing features using a parity setup and integrity streams.
is this a good idea or should try something else?
If you only have Windows then it is a fine option, but you do need a second drive and set up the storage in a mirror to take advantage of the automatic fixing with the backup copy. The auto detections and fixing doesnt just work on its own with no other parts to it. If you can use a Linux OS that has ZFS file system that is a better option as it handles bit errors more correctly.
1 Like
I was thinking about a 3 drive setup with parity for this.
Yes, ZFS is a great choice, if that’s not available BtrFS (with checksumming enabled) would also be an option. In both cases though, I believe that you need to have two copies of the files, either through a mirrored ZFS pool / BtrFS RAID or in the ZFS case through creating the filesystem in a manner, that it stores two copies of each file.
In any case I think that the mirrored pool/RAID is the preferred option, since it also protects you from drives failing.
In addition to a “reliable” file system you should also have at least two backups, one of which should be offsite (in case of fire, theft, etc.). An of course both of those should have a reliable file system as well. (I have a lot less important data, so my offsite backup is with Backblaze B2.)
Both with your primary storage and your secondary storage (assuming it’s not an online backup) you should run regular scrubs to ensure that the files are still accessible. (On my NAS, I run scrubs (and long SMART tests) every month, but for cold storage, you could perhaps go with an interval of two to three months.
ReFS doesn’t support Parity with the features you want, it only auto corrects with a spare copy on a mirror Storage Space. Unless the parity correction is something brand new, but as of when I was using ReFS 2 years ago I had the unfortunate experience of finding out that it only worked on a mirror. You can use a Parity SS, but you lose basically every benefit ReFS is supposed to be used for.
Realtime checksums in ReFS or ZFS detects damaged files. For an autorepair you need redundancy on both.
btw.
OpenZFS 2.3.0 on Windows is nearly ready with a new release every few days (currently rc5) so the pain with Windows filesystems or raid arrays or lacking features compared to ZFS can come to an end soon (while Storage Spaces with ntfs/ReFS to pool disks of different type or size with optional auto tiering is still superiour for this use case)
@gea By redundancy do you mean to say a storage space pool with parity or 2 separate full copies of the data on the same server?
With Storage Spaces redundancy (resilency) is not per Pool or disk but per individual Space (mirror or parity) to create additional data copies on this Space over different disks.
With ZFS redundancy is diskbased what meany every vdev in the pool must be a mirror, draid or raid-z