It appears as if I have a ZFS metadata inconsistency.
Situation:
I have a ZFS dataset (aka filesystem) which now refuses to unmount, have its mountpoint changed, or its content listed.
- A
umount
on path yields EINVAL which as I understand happens if the given path is not a mountpoint. - A
df
on the path shows the dataset and the supposed mountpoint but the stats of the parent filesystem.
(notice for the following output: /DataRecover
is a symlink to /Data
)
#showing actual stats and FS properties as they are supposed to be
$ zfs list Data/enc/DataRecover/media/music/yt
NAME USED AVAIL REFER MOUNTPOINT
Data/enc/DataRecover/media/music/yt 39.2G 2.87T 27.9G /DataRecover/media/music/yt.redownload/
#confirming with `mount`
$ mount | grep Data/enc/DataRecover/media/music/yt
Data/enc/DataRecover/media/music/yt on /Data/media/music/yt.redownload type zfs (rw,xattr,noacl)
#Notice the higher "Used" statistic
$ df /Data/media/music/yt.redownload/
Filesystem Size Used Avail Use% Mounted on
Data/enc/DataRecover/media/music/yt 3.2T 315G 2.9T 10% /Data/media/music/yt.redownload
#Shows no content on the mountpoint as if there's nothing mounted
$ ls /Data/media/music/yt.redownload
#trying to write to the supposedly mounted dataset
$ dd if=/dev/urandom of=/DataRecover/media/music/yt.redownload/test bs=1M count=4096
4096+0 records in
4096+0 records out
4294967296 bytes (4.3 GB, 4.0 GiB) copied, 2.27017 s, 1.9 GB/s
$ ls -lh /DataRecover/media/music/yt.redownload/
total 512
-rw-r--r-- 1 celmor celmor 4.0G Jan 21 03:36 test
#size unchanged
$ zfs list Data/enc/DataRecover/media/music/yt
NAME USED AVAIL REFER MOUNTPOINT
Data/enc/DataRecover/media/music/yt 39.2G 2.87T 27.9G /DataRecover/media/music/yt.redownload/
Operating on this mountpoint reveals inconsistencies:
$ zfs unmount Data/enc/DataRecover/media/music/yt
cannot unmount '/Data/media/music/yt.redownload': unmount failed
$ sudo umount /Data/media/music/yt.redownload
umount: /Data/media/music/yt.redownload: not mounted.
$ sudo strace umount /DataRecover/media/music/yt.redownload |& grep umount2
umount2("/Data/media/music/yt.redownload", 0) = -1 EINVAL (Invalid argument)
Or demonstrated using Linux native tools:
$ df /DataRecover/media/music/yt.redownload/
Filesystem Size Used Avail Use% Mounted on
Data/enc/DataRecover/media/music/yt 3.2T 315G 2.9T 10% /Data/media/music/yt.redownload
$ df /DataRecover/media/music
Filesystem Size Used Avail Use% Mounted on
Data/enc/DataRecover/media 3.2T 315G 2.9T 10% /Data/media
$ sudo strace umount /DataRecover/media/music/yt.redownload |& grep umount2
umount2("/Data/media/music/yt.redownload", 0) = -1 EINVAL (Invalid argument)
$ sudo mount --bind /DataRecover/media/music/yt.redownload/ link
$ df link
Filesystem Size Used Avail Use% Mounted on
Data/enc/DataRecover/media 3.2T 315G 2.9T 10% /tmp/target
Only bind mounting the supposed mount point somewhere else and quering that (via df
) reveals that it was never a mountpoint (giving me schroedingers cat vibes)
Further info:
$ zfs get encryptionroot Data/enc/DataRecover/media/music/yt
NAME PROPERTY VALUE SOURCE
Data/enc/DataRecover/media/music/yt encryptionroot Data/enc -
$ zfs get all Data/enc/DataRecover/media/music/yt | grep key
Data/enc/DataRecover/media/music/yt keylocation none default
Data/enc/DataRecover/media/music/yt keyformat passphrase -
Data/enc/DataRecover/media/music/yt keystatus available -
$ zfs get mounted,canmount Data/enc/DataRecover/media/music/yt
NAME PROPERTY VALUE SOURCE
Data/enc/DataRecover/media/music/yt mounted yes -
Data/enc/DataRecover/media/music/yt canmount on default
If there’s nothing I’ve missed I’m gonna open an issue on Github for this.