A ZFS Conundrum - Insights for Potential Data Recovery?

Software & Operating Systems Linux
, ,
1

Hi everyone, I am new to ZFS failure. I want to address my incident and wonder if there are any options available for my pool.

First of all, I’ve been using Truenas vm inside Proxmox, using by-id method.
Below are some of the basic information.

  • Truenas scale vm with 16 core, 64GB ecc reg, using tailscale only in app section.
  • Single raidz2 vdev, 10 x 16TB sata drives, around 1891 hours, no other vdev or special vdev.
  • smartctl short (weekly) passed, smartctl long (after incident) passed
  • No uncorrectable or pending sectors occurred while using Truenas
  • No sharing or remote service enabled (ssh, smb, nfs)
  • No power outages while using Truenas, running about 4 months straight until incident.

Incident happened about a week ago, all of a sudden my pool cannot be imported, I tried to export and import pool again inside Truenas.

And Truenas made an error, ValueError: 2095 is not a valid error.

ValueError: 2095 is not a valid Error

Error: concurrent.futures.process._RemoteTraceback: “”" Traceback (most recent call last): File “/usr/lib/python3.11/concurrent/futures/process.py”, line 256, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 112, in main_worker res = MIDDLEWARE.run(call_args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 46, in _run return self._call(name, serviceobj, methodobj, args, job=job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 34, in call with Client(f’ws+unix://{MIDDLEWARE_RUN_DIR}/middlewared-internal.sock’, py_exceptions=True) as c: File “/usr/lib/python3/dist-packages/middlewared/worker.py”, line 40, in call return methodobj(*params) ^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 181, in nf return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/plugins/zfs/pool_actions.py”, line 207, in import_pool with libzfs.ZFS() as zfs: File “libzfs.pyx”, line 529, in libzfs.ZFS.exit File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs/pool_actions.py", line 227, in import_pool zfs.import_pool(found, pool_name, properties, missing_log=missing_log, any_host=any_host) File “libzfs.pyx”, line 1369, in libzfs.ZFS.import_pool File “libzfs.pyx”, line 1397, in libzfs.ZFS.__import_pool File “libzfs.pyx”, line 658, in libzfs.ZFS.get_error File “/usr/lib/python3.11/enum.py”, line 717, in call return cls.new(cls, value) ^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3.11/enum.py”, line 1133, in new raise ve_exc ValueError: 2095 is not a valid Error “”" The above exception was the direct cause of the following exception: Traceback (most recent call last): File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 427, in run await self.future File “/usr/lib/python3/dist-packages/middlewared/job.py”, line 465, in __run_body rv = await self.method(([self] + args)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 177, in nf return await func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/schema/processor.py”, line 44, in nf res = await f(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3/dist-packages/middlewared/plugins/pool/import_pool.py", line 113, in import_pool await self.middleware.call(‘zfs.pool.import_pool’, guid, opts, any_host, use_cachefile, new_name) File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1399, in call return await self._call( ^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1350, in _call return await self._call_worker(name, prepared_call.args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1356, in callworker return await self.run_in_proc(main_worker, name, args, job) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1267, in run_in_proc return await self.run_in_executor(self.__procpool, method, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib/python3/dist-packages/middlewared/main.py”, line 1251, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, args, *kwargs)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ValueError: 2095 is not a valid Error

I’ve been poking my head for about a week and tried some of the options I found.

  1. zpool import -f shows ONLINE and make me promising, but there is no pool mounted.
zpool import -f & zpool status

zpool import
pool: hdd10x16t
id: 14448620205443767059
state: ONLINE
action: The pool can be imported using its name or numeric identifier.
config:hdd10x16t ONLINE
raidz2-0 ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZEA4 ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZD7P ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZESV ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZDE6 ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZE7G ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZDMX ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZD05 ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZED5 ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZEEA ONLINE
ata-ST16000NM001J-2TW113_ZRS0ZE3T ONLINE
zpool status
no pools available

  1. zpool import -f hdd10x16t returns insufficient replicas
zpool import hdd10x16t -f

zpool import hdd10x16t -f
cannot import ‘hdd10x16t’: insufficient replicas
Destroy and re-create the pool from a backup source.

  1. zdb -l hdd10x16t checked if by-id or devid changed. It returns not changed. (compared /dev/disk/by-id)
zdb -l hdd10x16t

zdb -l hdd10x16t

LABEL 0

version: 5000
name: ‘hdd10x16t’
state: 0
txg: 1268725
pool_guid: 14448620205443767059
errata: 0
hostid: 2285398396
hostname: ‘recode-hetzner10x16t’
top_guid: 3320680006278793367
guid: 9307791802452327806
vdev_children: 1
vdev_tree:
type: ‘raidz’
id: 0
guid: 3320680006278793367
nparity: 2
metaslab_array: 256
metaslab_shift: 34
ashift: 12
asize: 160008854568960
is_log: 0
create_txg: 4
children[0]:
type: ‘disk’
id: 0
guid: 9307791802452327806
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZEA4-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZEA4-part1’
phys_path: ‘pci-0000:25:00.0-ata-4.0’
whole_disk: 1
DTL: 159
create_txg: 4
children[1]:
type: ‘disk’
id: 1
guid: 5722856747687887882
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZD7P-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZD7P-part1’
phys_path: ‘pci-0000:01:00.0-ata-1.0’
whole_disk: 1
DTL: 158
create_txg: 4
children[2]:
type: ‘disk’
id: 2
guid: 12515852929378121397
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZESV-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZESV-part1’
phys_path: ‘pci-0000:01:00.0-ata-2.0’
whole_disk: 1
DTL: 155
create_txg: 4
children[3]:
type: ‘disk’
id: 3
guid: 5347527874058330893
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZDE6-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZDE6-part1’
phys_path: ‘pci-0000:01:00.0-ata-3.0’
whole_disk: 1
DTL: 154
create_txg: 4
children[4]:
type: ‘disk’
id: 4
guid: 16523393813715497135
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZE7G-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZE7G-part1’
phys_path: ‘pci-0000:01:00.0-ata-4.0’
whole_disk: 1
DTL: 153
create_txg: 4
children[5]:
type: ‘disk’
id: 5
guid: 846333880863613494
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZDMX-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZDMX-part1’
phys_path: ‘pci-0000:02:00.1-ata-1.0’
whole_disk: 1
DTL: 152
create_txg: 4
children[6]:
type: ‘disk’
id: 6
guid: 5875470405935328920
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZD05-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZD05-part1’
phys_path: ‘pci-0000:02:00.1-ata-2.0’
whole_disk: 1
DTL: 151
create_txg: 4
children[7]:
type: ‘disk’
id: 7
guid: 7378420708011026499
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZED5-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZED5-part1’
phys_path: ‘pci-0000:25:00.0-ata-1.0’
whole_disk: 1
DTL: 29336
create_txg: 4
children[8]:
type: ‘disk’
id: 8
guid: 12934111643194929302
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZEEA-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZEEA-part1’
phys_path: ‘pci-0000:25:00.0-ata-2.0’
whole_disk: 1
DTL: 25315
create_txg: 4
children[9]:
type: ‘disk’
id: 9
guid: 7616272590451776413
path: ‘/dev/disk/by-id/ata-ST16000NM001J-2TW113_ZRS0ZE3T-part1’
devid: ‘ata-ST16000NM001J-2TW113_ZRS0ZE3T-part1’
phys_path: ‘pci-0000:25:00.0-ata-3.0’
whole_disk: 1
DTL: 150
create_txg: 4
features_for_read:
com.delphix:hole_birth
com.delphix:embedded_data
com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3

  1. changed tvds and spa parameters and tried zpool import again. Shows same output
changing parameter and zpool import again

echo 1 > /sys/module/zfs/parameters/zfs_max_missing_tvds;
echo 0 > /sys/module/zfs/parameters/spa_load_verify_data;
echo 0 > /sys/module/zfs/parameters/spa_load_verify_metadata"

zpool import -o readonly=on hdd10x16t
cannot import ‘hdd10x16t : insufficient replicas
Destroy and re-create the pool from a backup source’

  1. passthrough sata controller with iommu and tried zpool import and other options above, shows same output. zdb -l hdd10x16t also gave the same output.

  2. Tries googled command while panicked -mfr, -fF readonly=on with pool name, but still shows insufficient replica.

After all of this, I wanted to ask if a couple of ideas I have in my head would work.

  1. As I think, zdb -l output shows create_txg : 4, So…Is there any way to properly grab the txg using zdb -e while my pool is unmounted?
    I watched the earlier video [TekSyndicate - Adventure in ZFS Data Recovery], but it seems risky, so if any of you have experience with this, I’d appreciate it if you could share.

  2. There is an article [Recovering Destroyed ZFS Storage Pools - Managing ZFS File Systems in Oracle® Solaris 11.2] In this article, it simply destroy and import the pool. it feels no way home, anyone tried this method?

  3. How’s your experience about [Klennet ZFS Recovery]? It feels GUI version of zdb -e script with a database,

  4. Or… Am I doing wrong?

If you have any questions about this, or if I can provide any information, please let me know. I’ll be happy to get right back to that.

1 Like
2

Are you sure the access to the disk from inside the VM works? I am far from an expert but if the virtual machine with Truenas can’t find a bunch of the disks it would explain why it refuses to import, because then in fact the replicas might not be sufficient.

Explicitly I am asking if zdb -l hdd10x16t lists the disks that are available or if it lists the disks that should belong to the array?

3

Inside Truenas,
I can access all of vdevs(disks) which I need, via Truenas. Truenas can run SMART test, shows status of I/O and temperature. And it has dev name starting ata-*, which is same with on the output of zdb -l hdd10x16t

4

What does zpool import -d /dev/disk/by-id/ return?

And what does zpool import -D return?

5

Hi, Thanks for your feedback,
here are output of each.

zpool import -d /dev/disk/by-id/

zpool import -d /dev/disk/by-id/

sudo zpool import -d /dev/disk/by-id/
pool: hdd10x16t
id: 14448620205443767059
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and
the ‘-f’ flag.
see: Message ID: ZFS-8000-EY — OpenZFS documentation
config:

hdd10x16t                                    ONLINE
  raidz2-0                                   ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZEA4-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZD7P-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZESV-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZDE6-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZE7G-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZDMX-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZD05-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZED5-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZEEA-part1  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZE3T-part1  ONLINE

zpool import -D

zpool import -D

sudo zpool import -D
no pools available to import

1 Like
6

What exactly does zpool import -f hdd10x16t output? Like did you mix up the name and the parameter further up?

You could also try zpool import -F hdd10x16t!

7

I’ve tried that command but retried once more, after -d and -D command
Here are the output

zpool import -f hdd10x16t

zpool import -f hdd10x16t

sudo zpool import -f hdd10x16t
[sudo] password for admin:
cannot import ‘hdd10x16t’: insufficient replicas
Destroy and re-create the pool from
a backup source.

zpool import -F hdd10x16t

zpool import -F hdd10x16t

sudo zpool import -F hdd10x16t
cannot import ‘hdd10x16t’: insufficient replicas
Destroy and re-create the pool from
a backup source.

8

Sorry, but I am out of my depth here. It seems to me that your pool is in fact lost. This message about the replicas seems to indicate that there was some kind of failure and ZFS does not have needed parity available to fix the problem. But maybe someone else has a different insight.

1 Like
9

Thank you for your time and effort.

Trying to digging whole internet as best as I can (even with several LLMs), so far I couldn’t find a way out. Everything I’ve found seems to assume that the pool is imported, and I was too afraid to try the 8 year old video from Wendell’s old channel, Tek Syndicate, that I mentioned earlier. I also decided that modifying the parameter of Truenas scale itself was too much of a risk.

I’ll going to try Klennet ZFS Recovery, crossing all my fingers.
There’s probably no case on the internet for using Klennet ZFS Recovery specifically for non-importing pools, so I’ll update this post after I attempt the recovery.

1 Like
10

Only idea I still have is that maybe there is an issue with ZFS itself. You might try to start another Live ISO or Distribution and try to import the pool there. Maybe your current System has an issue with the cache file or something that is not apparent.

11

@honey_combo how often were scrubs run on the pool?
It’s possible (however seems unlikely) that ZFS let too much bitrot set in and became un-importable.

12

Thanks, but seems not imported because featureflag.

I just made a ubuntu 22.04 VM and passthrough-ed SATA controllers (iommu).

lsb_release -a

No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 22.04.4 LTS
Release: 22.04
Codename: jammy

and installed zfs using sudo apt install zfsutils-linux

zfs version 
zfs-2.1.5-1ubuntu6~22.04.3
zfs-kmod-2.1.5-1ubuntu6~22.04.3

And here it is the output of zpool import and told to me unsupported feature(s)

sudo zpool import

sudo zpool import
pool: hdd10x16t
id: 14448620205443767059
state: UNAVAIL
status: The pool was last accessed by another system.
action: The pool cannot be imported due to damaged devices or data.
see: Message ID: ZFS-8000-EY — OpenZFS documentation
config:

hdd10x16t                              UNAVAIL  unsupported feature(s)
  raidz2-0                             ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZEA4  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZD7P  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZESV  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZDE6  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZE7G  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZDMX  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZD05  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZED5  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZEEA  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZE3T  ONLINE

I think maybe from below, any ideas to ignore this?

zdb -l hdd10x16t, last 4 lines 
features_for_read:
    com.delphix:hole_birth
    com.delphix:embedded_data
    com.klarasystems:vdev_zaps_v2
labels = 0 1 2 3
13

Hi @twin_savage, it scheduled every month and passed scrub with 0 errors since this pool created.

The reason I created this zfs pool was to act as a buffer before moving Google Drive unlimited storage to local, as Gdrive ends around December 15th.
So I assume that all write operations to the zfs pool have been done at least two months ago. Also there is no log reported about error(scrub or pending sector, etc.) on Truenas.

14

ahhh, then that is exceedingly unlikely scenario then, can be outright dismissed.

I still like the idea of trying to import the pool into a “fresh” VM. it would (or could) rule out an issue with the disks vs ZFS configuration.

15

Yes I think it is great idea from @H-i-v-e and I tried. but it seems blocked by feature flag.

16

You could try another OS that has a matching ZFS version with your Truenas ZFS version to get around that problem.

17

Thanks for your feedback.

As I see on version on the pool using zdb -l hdd10x16t, it shows version: 5000
Is it a thing I should resolve for? I thought ubuntu VM’s problem is only from feature_flag

EDIT: there is a difference version on Truenas vs Ubuntu
Truenas zfs verson is zfs-2.2.3-1, zfs-kmod-2.2.3-1
Ubuntu zfs version is zfs-2.1.5-1ubuntu6~22.04.3, zfs-kmod-2.1.5-1ubuntu6~22.04.3. Note that I followed install instruction on Getting Started — OpenZFS documentation

18

oh, I was making the possibly incorrect assumption you were using features in your pool that were not supported on ubuntu’s version of zfs.
There should be feature arguments for the import command that can address the issue, but it is still likely most simple to import the pool to a system running the same version of zfs.

1 Like
19

Thanks for your time @twin_savage .
I found out Fedora can install zfs 2.2.3-1.
following the guide Fedora — OpenZFS documentation and installed successfully zfs 2.2.3-1

but it makes same output before

zfs version, zpool import and with parameters

uname -r

6.6.22-200.fc39.x86_64

zfs version

zfs-2.2.3-1
zfs-kmod-2.2.3-1

sudo zpool list

no pools available

sudo zpool import

 pool: hdd10x16t
   id: 14448620205443767059
state: ONLINE
status: The pool was last accessed by another system.
action: The pool can be imported using its name or numeric identifier and	the '-f' flag.
see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-EY
config:

hdd10x16t                              ONLINE
  raidz2-0                             ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZEA4  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZD7P  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZESV  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZDE6  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZE7G  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZDMX  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZD05  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZED5  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZEEA  ONLINE
    ata-ST16000NM001J-2TW113_ZRS0ZE3T  ONLINE

sudo zpool import -D

no pools available to import

sudo zpool import -d /dev/disk/by-id

SAME OUTPUT with zpool import

sudo zpool import -fd /dev/disk/by-id

SAME OUTPUT with zpool import

sudo zpool import -f -o readonly=on hdd10x16t

cannot import 'hdd10x16t': insufficient replicas
Destroy and re-create the pool from
a backup source.

So… I think I have two options at minimal risk, one is Klennet ZFS Recovery the other is Recovery angel:데이터복구 프로그램 [Standard RAID] 제품 안내

I’m going to try Klennet first, I expect it takes 3 days to complete 2nd scan.
I’ll let all know when it complete scan.

1 Like
20

let us know how klennet turns out, I really enjoy reading the blog kept there.