[solved] ZFS: Monthlong changes to dataset gone

Operating Systems & Open Source
,
1

Hi,

I screwed up my zfs setup and now face a month long data loss due to that (Yeah, no backup, my fault, I know)

I have a small zfs pool in a mirror configuration with an encrypted dataset. This dataset is shared via smb and was used heavily over the last months. Today I had to perform a reboot of the server (the first in months) and after that the dataset is as the last couple of months haven’t had happened. Folders and files are missing.

The pool was scrubed regularly and everything was healthy. Since this is not per-se a dataloss due to faulty hard drive(s) I am rather lost on how (and if ) I can recover that data. Help is very much appreciated.

Note:

  • Somehow the dataset didn’t mount automatically during boot
  • When not mounted, some sub datasets seem to carry data which vanishes when the zfs dataset is mounted
2

Since you suspect data-loss, you should do the following ASAP.

  1. Umount ZFS array.
  2. Take copies of drives (using dd) onto spare drives.

After that it sounds like you may have experienced a common UNIX issue - mounting over a directory hides data in that directory. If you had a ZFS array that mounted at /foobar, you may have accidentally written files to /foobar before mounting the zfs array. You can use a bind mount to see all the files on / to see if the files are present.

mkdir /temp-mount
mount -o bind / /temp-mount
ls -la /temp-mount/foobar
# Repeat bind mount for all filesystems

If this doesn’t help, then you might need to investigate ZFS to see if an issue occurred. You could try mounting the drive copies one at a time. And if that doesn’t help and you run out of options, try using dark magic to roll back ZFS transactions. (Don’t do any of these on your production array, only on copies).

However from what you’ve presented, I don’t see a reason to suspect ZFS data loss at this stage.

1 Like
3

Hi, thx for your answer. I agree the “write-to-non-mounted dir” happened. I can clearly see that in the directories which vanish when the zfs dataset is imported.

The strange thing is:
For the specific directory that started this post, this is not the case. Here the directory is completely empty and only when the dataset is imported, the old data appears. But up to the last reboot it was both, the new and the “old” data.

Thus, I am not sure how to proceed. Assuming I can repair things with transaction magic, I still can’t be 100% sure that the data will be complete. I guess I have to re-do the whole work anyway. Besides: How can one avoid this “not-mounted” problem? I can not rely on me remembering to check if the zfs mount was a success after a reboot. Any ideas? (so that I at least learn something from the desaster)

4

Thus, I am not sure how to proceed.

Did you use the bind mount trick to check if the files are on your root volume? What about ZFS datasets, do you have any on your pool?

As for ZFS recovery, without knowing why there may have been corruption I’m not sure what to suggest. As a start you could do some debugging with these tools:

zpool history pool
zpool events pool

How can one avoid this “not-mounted” problem?

That’s actually incredibly easy, put everything into subdirectories on your drive. Though this can depend on your program.

Let’s say you have a mountpoint as “/spa”. If you write files to “/spa”, it will be able to write them even when the drive disconnected. However if you create a subdirectory “/spa/jet” on your ZFS disk, programs will be unable to write to the “/spa/jet” directory while your disk is unmounted. This entirely avoids your problem. So always put things in sub directories on your mounted drives.

There is an exception, some programs will happily create subdirectories if they don’t exist. In this case there are a few ways to tackle it.

  • Permissions. If your program operates as a user, simply set the mount directory to root permissions, while the drive isn’t mounted. Then your program can’t create subdirectories.
  • Use a shell script to check if the directories are mounted. I’ve used something like Example A in a few of my scripts. You could also do a “-d” check for the subdir.

Example A:

if ! /bin/findmnt /spa --raw -n -o target > /dev/null; then
  echo "spa isn't filled"
  exit 1
fi
1 Like
5

Thx for your insight, I will try to set it up that way next time.

> Here was a full description of my current setup. After writing the last paragraph, I noticed my issue.

Okay, everything is in order, the data is intact. My dataset structure is the following

/bucket <- mounted at boot
/bucket/media <- encrypted
/bucket/media/foo

  • After the reboot, only /bucket and non encrypted datasets are mounted (/bucket/media/foo is empty)

  • Hence I added the key and mounted /bucket/media (/bucket/media/foo partially populated)

  • What I didn’t do (because I am dumb) is to mount /bucket/media/foo . Doing that gives me the rest of the data

Hence, my guess now is that during one of the last reboots I missed to mount the whole dataset chain, leaving the sub/sub datasets unpresent, but mountable via smb.

This issue affects also the rest of /bucket/media/* datasets. Also this data shouldn’t be encrypted right now, which needs to be fixed. I wonder if there is a way to move data from one dataset to the next ? (need to look that up)

Thank you for your help and sorry I wasted your time.

1 Like
6

Use the mv command to move data out of unmounted mount point, mount dataset, then mv into dataset.