Backing up Linked Data on NAS the right way

Hi All

We’re a 10 person 3D studio using Blender all of our renders. We are planning our transition from “packed” .blend files to “linked” .blend files. (Packed houses all images, textures,… inside of the same .blend file; Linked just stores the metadata and the network path to the images, textures,…)

The reasons for this are obvious: our blend-files are becoming larger and larges as our projects are more ambitious. Where a typical project used to be 700MB, we now have projects of +3.5GB. Fixes to a furniture model for instance are now done on a project bases instead of fixing it ‘upstream’ in our library.

Now the problem comes in long term storage of our projects. It happens that a client comes back after 2 or 3 years and asks to make changes to his/her project. We want to be able to re-use the original blend file from years back. Linking files makes it way harder for us to make changes to our Library(structure). For instance, in case we’d want to make the choice to migrate our Library, currently mapped as V:, to a more logical L:\ on our workstations all our links break. Let alone the implications if we’d want to overhaul our folder structure. Quite frankly I’d like our system to robust enough to survive when our Library would suddenly be destroyed and no backup of that could be restored.

My solution for this was to have have our ‘hot’ NAS backup its files to our ‘cold’ off-site vault and letting it pack (and compress) everything right before sending it off.
Last week I built a new NAS system (Freenas) that uses 10Gbps instead of our 1Gbps windows file share we’re were using. So I’ve got a enough CPU headroom on our NAS to do this.

While I hardly think it’s best practice to operate on data before moving it to cold storage, I’m failing to find a better solution. Another problem stemming from this approach is that common backup solutions that use rsync for instance will fail since the checksums won’t match.

It would require me to write my own backup script that makes a checksum of the hot/linked version; packs & compresses everything; sends it off to the vault. Next run it would just check against the checksum of the hot version to see if anything changed.

Is there a best practice for this scenario? Would this be a ‘fine’ approach? Any other tips or pointers on how to make this a robust system?

Zfs has some built in stuff to handle this. Check out zfs send. Other than that checksumming as you’re doing is what’s required.

You might be able to automate rsync though? That scans for changes and will sync them automatically. It works well.

Zfs send doesn’t have to do checking per se… it knows because it just syncs up the blocks that have a newer serial number than the last zfs send so it’s pretty cool in that way

It’s great for snapshots, accidental deletion recover and being extremely efficient with over the wire copies

1 Like