Need some help with zfs errors in truenas

Hardware Hub
, , , ,
1

Hi all, trying to figure out these ZFS errors I’m seeing in truenas and wanted to ask for advice here. I have 5 4TB ST4000NM0033-9ZM170 plugged into a HBA off aliexpress (INSPUR 9300-8I). These drives are… well used, they have about 8 years of power on time. They all pass SMART, at least I read it that way. The HBA is plugged into a Lenovo m920q with the riser card option. At first I thought it was a power problem since I have 4 on one external 12v PSU and the other 4 on another PSU, one does 10a and one does 8a (slightly overloaded in terms of spec for 4 drives). An example of this was sdc and sdd both said they faulted, but when I switched the power cable to the more beefy psu, the faults went away entirely on one drive, the other drive had 2 checksum errors. The image below shows what it looks like now after being on the properly spec’d PSU, before it had Write errors.

image
image486×307 5.25 KB

image
image403×488 92.5 KB

The current drive array looks like this where each side has its own PSU:

Left Right
HDD SSD
HDD SSD
HDD
HDD HDD

The drive that has the checksum errors is on the left.

Here’s the smartctl for the drives.

SDA

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   063   044    Pre-fail  Always       -       187357237
  3 Spin_Up_Time            0x0003   096   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       102
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       1192409712
  9 Power_On_Hours          0x0032   022   022   000    Old_age   Always       -       68729
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       102
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       1
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   059   042   040    Old_age   Always       -       41 (Min/Max 37/41)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       98
193 Load_Cycle_Count        0x0032   022   022   000    Old_age   Always       -       157013
194 Temperature_Celsius     0x0022   041   058   000    Old_age   Always       -       41 (0 18 0 0 0)
195 Hardware_ECC_Recovered  0x001a   022   013   000    Old_age   Always       -       187357237
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Self-test routine in progress 90%      3193         -
# 2  Extended offline    Completed without error       00%      3131         -
# 3  Extended offline    Completed without error       00%      2974         -
# 4  Short offline       Completed without error       00%      2966         -
# 5  Extended offline    Interrupted (host reset)      90%      2957         -
# 6  Short offline       Completed without error       00%      2885         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testin

SDB

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   073   063   044    Pre-fail  Always       -       21368519
  3 Spin_Up_Time            0x0003   094   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       88
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       1113438518
  9 Power_On_Hours          0x0032   022   022   000    Old_age   Always       -       68688
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       89
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   095   095   000    Old_age   Always       -       5
190 Airflow_Temperature_Cel 0x0022   062   043   040    Old_age   Always       -       38 (Min/Max 34/38)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       86
193 Load_Cycle_Count        0x0032   021   021   000    Old_age   Always       -       158798
194 Temperature_Celsius     0x0022   038   057   000    Old_age   Always       -       38 (0 18 0 0 0)
195 Hardware_ECC_Recovered  0x001a   013   012   000    Old_age   Always       -       21368519
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Self-test routine in progress 90%      3152         -
# 2  Extended offline    Completed without error       00%      3089         -
# 3  Short offline       Completed without error       00%      2884         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing

SDC

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   078   063   044    Pre-fail  Always       -       62902161
  3 Spin_Up_Time            0x0003   093   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       95
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       1117804576
  9 Power_On_Hours          0x0032   022   022   000    Old_age   Always       -       68657
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       94
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       1
189 High_Fly_Writes         0x003a   096   096   000    Old_age   Always       -       4
190 Airflow_Temperature_Cel 0x0022   061   046   040    Old_age   Always       -       39 (Min/Max 33/39)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       92
193 Load_Cycle_Count        0x0032   021   021   000    Old_age   Always       -       158666
194 Temperature_Celsius     0x0022   039   054   000    Old_age   Always       -       39 (0 18 0 0 0)
195 Hardware_ECC_Recovered  0x001a   021   013   000    Old_age   Always       -       62902161
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Self-test routine in progress 90%      3121         -
# 2  Extended offline    Interrupted (host reset)      00%      3054         -
# 3  Extended offline    Interrupted (host reset)      00%      2897         -
# 4  Extended offline    Interrupted (host reset)      90%      2888         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing

SDD

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   077   063   044    Pre-fail  Always       -       50751895
  3 Spin_Up_Time            0x0003   094   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       93
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       1118231216
  9 Power_On_Hours          0x0032   022   022   000    Old_age   Always       -       68667
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       99
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   092   000    Old_age   Always       -       8
189 High_Fly_Writes         0x003a   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   059   045   040    Old_age   Always       -       41 (Min/Max 36/41)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       69
193 Load_Cycle_Count        0x0032   024   024   000    Old_age   Always       -       153763
194 Temperature_Celsius     0x0022   041   055   000    Old_age   Always       -       41 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   017   014   000    Old_age   Always       -       50751895
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Self-test routine in progress 90%      3131         -
# 2  Extended offline    Interrupted (host reset)      00%      3066         -
# 3  Extended offline    Completed without error       00%      2912         -
# 4  Extended offline    Interrupted (host reset)      90%      2895         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing

SDG

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   084   063   044    Pre-fail  Always       -       237154862
  3 Spin_Up_Time            0x0003   093   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       94
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   090   060   045    Pre-fail  Always       -       1165280396
  9 Power_On_Hours          0x0032   022   022   000    Old_age   Always       -       68735
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       94
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
189 High_Fly_Writes         0x003a   099   099   000    Old_age   Always       -       1
190 Airflow_Temperature_Cel 0x0022   060   047   040    Old_age   Always       -       40 (Min/Max 36/40)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       91
193 Load_Cycle_Count        0x0032   022   022   000    Old_age   Always       -       157154
194 Temperature_Celsius     0x0022   040   053   000    Old_age   Always       -       40 (0 17 0 0 0)
195 Hardware_ECC_Recovered  0x001a   024   011   000    Old_age   Always       -       237154862
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Self-test routine in progress 90%      3199         -
# 2  Extended offline    Completed without error       00%      3137         -
# 3  Extended offline    Completed without error       00%      2979         -
# 4  Extended offline    Interrupted (host reset)      90%      2962         -
# 5  Short offline       Completed without error       00%      2884         -

I would love some ideas on if the drives are usable or what the issue may be.

2

From a quick look I don’t see anything worrying there. (Just FYI, those big scary numbers like “Hardware_ECC_Recovered” are actually fine, Seagate just packs two numbers into one value).

What I’d so is ensure you have backups, perform a scrub, then keep using your pool and monitoring ZFS error counts. Smart is about “predictive” failures, “I think this will fail”. The best way to tell if a drive is failing is … honestly zfs error counts. So keep using them and see you in five years.

3

The power issue is the first thing I’d investigate, especially since moving the drives to the stronger PSU made the faults disappear. SMART looks reasonably clean for these drives, but the ~68,000 hours of runtime is significant. I’d keep monitoring ZFS checksum/IO errors and test the HBA, cables, and power delivery one at a time before trusting the pool. I wouldn’t treat a clean SMART test as proof that the drives are completely healthy.

4

check zpool status -v

UI stuff from TrueNAS sadly lacks a lot of important stuff.

If there are no instructions for proceeding are given by zpool status, like restore from backup or corrupt files listed, I wouldn’t worry. Especially not when the culprit has been found and fixed since then.

There is a command to reset the error counters, but I can’t remember it anymore.

2 errors isn’t that much, especially if it’s just one drive and ZFS can self-heal on its own. If a file or two are corrupted, deleting those files will clear the error too…

If there are errors regularly and frequent, ZFS will lockdown the pool for you and throw I/O errors. As long as you can import the pool and write to it, everything’s fine with your data :slight_smile:

5

All, thank you for the input. This is a new install on these drives, so no data is lost luckily. The plan is for this to be a NVR with frigate.

Here is the zpool status. My next plan it to grab a higher rater PSU so that both are 10a and continue testing. As suggested below, I will clear errors and see if it reappears after writing some junk data to it.

errors: No known data errors

  pool: nvr-pool
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-9P
  scan: resilvered 17.9M in 00:00:01 with 0 errors on Sun Sep 27 13:16:00 2026
expand: expanded raidz2-0 copied 3.88G in 00:00:25, on Sun Sep 20 04:41:19 2026
config:

        NAME                                      STATE     READ WRITE CKSUM
        nvr-pool                                  ONLINE       0     0     0
          raidz2-0                                ONLINE       0     0     0
            03c9269c-12d9-482e-885c-e1860cdaee48  ONLINE       0     0     0
            306eef6f-0b2a-4c04-8cab-80e8ba2727d6  ONLINE       0     0     0
            b173596c-0e64-41c9-8890-05ee4b18343b  ONLINE       0     0     2
            e2b25ffa-0b37-4e70-b7b0-c1bf01b28b79  ONLINE       0     0     0
            14022598-fd4b-49b0-a487-4522029b5618  ONLINE       0     0     0
6

all fine. I’d go for the zpool clear option. Error was self-healed by ZFS thanks to your redundancy (Z2)

e.g. zpool clear nvr-pool

1 Like