Hi there! I am a super ultra noob to ZFS and TrueNAS. I am storing photos and old documents and things in my NAS, and I have been wanting to upgrade. After getting a new Fractal Define 7 case and a larger power supply, I moved the six 4TB Seagate Iron Wolf disks over to the new case. My pool is in raid Z2.
After running a scrub, my pool was marked DEGRADED, and one of the drives is marked as faulted. Here is the data:
root@truenas[~]# zpool status -v
pool: nargothrond
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
Sufficient replicas exist for the pool to continue functioning in a
degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
repaired.
scan: scrub repaired 1.72M in 01:00:02 with 0 errors on Sun Oct 9 07:31:24 2022
config:
NAME STATE READ WRITE CKSUM
nargothrond DEGRADED 0 0 0
raidz2-0 DEGRADED 0 0 0
948e0e11-b19a-467c-a273-7c8f77943134 ONLINE 0 0 0
a15c7c1a-3a40-4229-aa87-a711bc65c738 FAULTED 59 9 0 too many errors
0ade8e27-a01b-48bf-8dab-744624cae90e ONLINE 0 0 0
5edd8b59-3128-463f-83db-f443fd0cbc31 ONLINE 0 0 0
5984f284-37f4-4693-9911-d0c5c0b122a0 ONLINE 0 0 0
157cd5f5-8e26-4982-a861-cec077fccbc1 ONLINE 0 0 0
errors: No known data errors
Now, I am a noob, so I did not know what to do with this information except run an extended SMART test through the web GUI. I did that, and I could have sworn a got a failure result after many hours. But today, after I have started learning about smartctl, the results show passing, and I donāt know why SMART would show failing one day and passing the next.
I am now re-running an extended test on the drive to get the CLI data, so you will see that here:
root@truenas[~]# smartctl -a /dev/sdf
smartctl 7.2 2020-12-30 r5155 [x86_64-linux-5.10.142+truenas] (local build)
Copyright (C) 2002-20, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate IronWolf
Device Model: ST4000VN008-2DR166
Serial Number: WDH222B8
LU WWN Device Id: 5 000c50 0a84cb267
Firmware Version: SC60
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5980 rpm
Form Factor: 3.5 inches
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-3 T13/2161-D revision 5
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Mon Oct 10 19:26:17 2022 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 249) Self-test routine in progress...
90% of test remaining.
Total time to complete Offline
data collection: ( 591) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 640) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x50bd) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 083 064 044 Pre-fail Always - 180718406
3 Spin_Up_Time 0x0003 093 093 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 69
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 081 060 045 Pre-fail Always - 126477149
9 Power_On_Hours 0x0032 098 098 000 Old_age Always - 1893 (122 189 0)
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 69
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 099 000 Old_age Always - 17180131332
189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 070 054 040 Old_age Always - 30 (Min/Max 27/30)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 10
193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 99
194 Temperature_Celsius 0x0022 030 046 000 Old_age Always - 30 (0 16 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 9
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 1891h+18m+26.042s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 34633819266
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 33357568627
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Self-test routine in progress 90% 1893 -
# 2 Extended offline Completed without error 00% 1865 -
# 3 Extended offline Interrupted (host reset) 00% 1851 -
# 4 Extended offline Completed without error 00% 1695 -
# 5 Extended offline Completed without error 00% 1528 -
# 6 Extended offline Completed without error 00% 1359 -
# 7 Extended offline Completed without error 00% 1191 -
# 8 Extended offline Completed without error 00% 1023 -
# 9 Extended offline Completed without error 00% 946 -
#10 Extended offline Completed without error 00% 819 -
#11 Extended offline Completed without error 00% 650 -
#12 Extended offline Completed without error 00% 525 -
#13 Extended offline Interrupted (host reset) 00% 483 -
#14 Extended offline Interrupted (host reset) 00% 481 -
#15 Extended offline Completed without error 00% 415 -
#16 Extended offline Interrupted (host reset) 80% 406 -
#17 Extended offline Completed without error 00% 306 -
#18 Extended offline Interrupted (host reset) 60% 218 -
#19 Extended offline Completed without error 00% 177 -
#20 Extended offline Completed without error 00% 79 -
#21 Conveyance offline Completed without error 00% 13 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
As you can see, I have a lot of āInterrupted (host reset)ā failures. I checked the other drives, and I saw similar SMART failures, except for the most recent one at the top of this faulted one.
There was a time when I was configuring NUT and testing that TrueNAS would power down correctly, but I donāt remember ever shutting down the NAS in an ungraceful manner. I am not sure what these errors mean.
So now I am waiting for another extended to test to complete around 5am tomorrow. I donāt have any spare drives at the moment, but I just bought two more 4TB Iron Wolf drives on eBay and they are coming within the next few days along with some additional drive caddies.
Should I consider the drive faulted? What would you do in this situation? Do you have any advice for me?