TRUENAS - My first data-boo-boo (deleted dataset, but recovered luckily)

Hey all,

Thought I’d share my mistakes, as well as my wins :roll_eyes:

I have 3 TrueNAS servers, 1 sync’s (Replication) to 2, and 2 sync’s with 3 for my daily Pool. Then I have my main ‘at rest’ pool, that sync’s between 2-3. only.

So I was on Server 2, thinking I was on Server 3…aaaaand deleted an important work dataset (70GB). Now I did have a copy on my old Windows machine (using backblaze for backup), the thing I used before proper data care, but it was going to be out of date and missing around 5TB of data I’d prefer not to lose.

Panic set in, then just as I was booting up my old windows machine, I had the light bulb moment. I don’t run a snapshot/replication server for nothing. Logged in, most recent replication was only a few hours before and hadn’t touched the files for weeks.

Restored files in about 10-15mins, wow. That’ll teach me. All because I wasn’t looking at the IP address properly. To address the user issue, I think I’m going to change the GUI theme or something, to avoid it happening again!

I have to say, TrueNAS is a brilliant thing…shame about the user :joy:

6 Likes

Thank you for sharing. I am getting ready to set up a remote TruNAS for offsite backup and replication.

2 Likes

You’re very welcome - if you could share some info about how you do that, I’d be very interested. VPN’s things like that is something I haven’t tinkered with as yet…in fear of leaving myself open to bad people!

I was going to use a wireguard tunnel to connect them.

That’s what I’d heard was good to use - or talescale. The latter confused me when I tested it though!

I think PFSense uses/has wireguard, but I’ve also seen references to OpenVPN in the services area of TrueNAS.

Normal people delete files to get into troubles, ZFS users have to delete entire datasets to feel the same. I guess everyone, regardless of their choice of storage, manages to get into trouble :slight_smile:

There are always recovery options like TXG rollbacks of the entire pool. But I’m glad you had a backup without the need for desperate measures.

RAID is not a backup. Humans make mistakes. Thanks for confirming this wisdom.

But good to see you’re stress testing your new setup. Seems like your homelab can account for your deficits :wink:

3 Likes

:laughing: :laughing: :laughing: That made me laugh, yeah ZFS users do it properly!

You’re quite right, I could have used rollback or gone from pull to push, but panic of the moment made me do what was simplest for my brain to comprehend :slight_smile:

+1 for sure, and you’re welcome (though I’d have preferred if it was someone else.

I’m stress testing me! :laughing: When I’m brave, I might try leaving a very low power TrueNAS server at my parents (live in the same village) so I can have a proper remote backup. The beauty of it is that I’ll be able to do the majority of the data on a LAN, and then drop it off to them. Hmm, actually I could play with doing that by using a spare machine, loading TN and leaving it round there turned on. Thanks for the light bulb moment :+1:

Yeah, my router is pfSense so I was just going to set up the server there. I am actually just going to be leasing some compute and storage on a friends T430 in exchange for some IT work. I am going to virtualize TruNAS Scale on it and pass through some HDD’s I am sending him and then I will have a offsite backup for the cost of the drives. His house has symmetrical 1 gig fiber that doesn’t get used as much as it should lol.

Wow, that’s an ideal situation eh, I’ll be very interested to know how you get on, especially virtualising Scale. You probably know, but visiting the truenas forum, there are lot of tales of woe about virtualising it.

1gig symmetrical fibre though, me want! :laughing:

First replication takes bandwidth, but if you send incremental snapshots on a e.g. daily basis, it’s very manageable even without symmetric fiber WAN.

I don’t really see the point in virtualizing Scale unless you’re desperate for nested containerization. If you just need a storage server, get Core.

1 Like

What are the tales of woe? I am virtualizing it rn and I haven’t had any issues.

I love a good “I deleted some data accidentally and recovered in full” story!

Now I’m sure we all found ourselves in a situation such as this one, and I think a good solution is ability to apply color to some UI element out of the box. I’ve asked QNAP and Synology to make something like it but got zero response.

Just imagine on TrueNAS a green sidebar (go ahead, do whatever) for your production server, yellow (be careful) for primary backup and red (last resort, be super careful) for secondary backup. Always visible, big UI element to keep you alert.

I should go bug TrueNAS people about it next.

2 Likes

Are are 6+ different themes (one light, one dark and dracula/solarized for 3rd server?) for TrueNAS and I’m pretty sure you can make your own if you tweak some text files. And for multiple Servers, there is TrueCommand.

1 Like

I know, I just meant like consistent UI with a single element to highlight where you are. Am I just weird?

:joy: Glad my sweaty moment entertained!

I really like your thinking about colours (red, amber, green). I’ve just gone with the optional colours right now. It would even be nice to have the server name a bit bigger (top right).

I’ve created a thread that shows theme options, if anyone doesn’t want to install TrueNAS to look for themselves.

I did want to get TrueCommand, but I don’t have any containery servers :frowning: I do have a Pi that’s only running Pi-Hole, I wonder if I can put it on that?

Hmm, I’m inclined to get a super low power / cheap old machine and bung Scale on that, then try Command. :slight_smile: