(SOLVED) BtrFS -> NTFS transfer speeds seem notably slow

Software & Operating Systems Linux
,
1

So, I have two 4TB HDDs in a server I use. One is formatted as a single BtrFS partitionless disk. The other is formatted as NTFS, as I intend to move it to a Windows computer.

I'm currently copying everything from the BtrFS to the NTFS drive. My speeds are 20 MB/s at maximum, and with smaller files, it drops to sub-5 MB/s (these are 3-5MB size files).

The BtrFS drive is not duplicating data, but is duplicating metadata. It is a WD Red Pro (7200RPM) drive.

The NTFS drive used default settings. It is a Seagate 4TB 5900RPM drive.

Is this type of performance normal? I'm personally used to transfers at >40 MB/s using HDDs in this server, but this is the first time I've copied between the two drives in this situation (BtrFS to NTFS).

And if it's normal, which is the issue? BtrFS or NTFS? According to SMART information, neither drive is notably hot, nor has any bad sectors.

I'm using rsync -ah to copy the files.

2

What is the specs of your system?

Any reason your using rsync to copy data instead of cp?

3

are they both sata? or is one USB?

4

Intel Atom Avoton CPU - 8 core 2.0 GHz
SuperMicro M-ITX Motherboard:

16GB of ECC Kingston DDR3 RAM (laptop form factor)

@SgtAwesomesauce They are both SATA, according to their hdparm information, they're both using SATAIII (I realize they can't do 6.0 Gbps, let alone 1.5 Gbps, but still).

Dje4321, I took from this:

IMO: alias cp="rsync -avz" cp is outdated – Marco Ceppi♦

cp lacks a progress output without using another package. This is copying 2.42TiB of data.

Also: http://serverfault.com/questions/43014/copying-a-large-directory-tree-locally-cp-or-rsync

It is a large directory tree with many large and small files. It pushes a near constant 19.5ish MB/s during large (multi-GB) files.

I want to point out I'm not using -z flag for rsync. Compressing for local transfers is unnecessary overhead and primarily for network transfers (as it mentions in that askubuntu thread).

According to htop, none of my CPU cores are pegged at 100% usage and RAM is sitting at less than 25% usage.

I'd like to point out that iostat reported around 19-20 MB/s transfer speeds when I was trying cp to start with.

5

Smart tests are all good? How old are the drives? What's the model number on them. It's possible you have a bad drive.

6

Obtained using hdparm -I:

/dev/sda:

ATA device, with non-removable media
        Model Number:       WDC WD4001FFSX-68JNUN0
        Serial Number:      WD-WCC5D1ZCNKK4
        Firmware Revision:  81.00A81
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Supported: 8 7 6 5
        Likely used: 8
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:    16514064
        LBA    user addressable sectors:   268435455
        LBA48  user addressable sectors:  7814037168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:     3815447 MBytes
        device size with M = 1000*1000:     4000787 MBytes (4000 GB)
        cache/buffer size  = unknown
        Nominal Media Rotation Rate: 7200
Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 32
        Standby timer values: spec'd by Standard, with device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Advanced power management level: 128
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    NOP cmd
           *    DOWNLOAD_MICROCODE
           *    Advanced Power Management feature set
                Power-Up In Standby feature set
           *    SET_FEATURES required to spinup after power up
                SET_MAX security extension
           *    48-bit Address feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
                Media Card Pass-Through
           *    General Purpose Logging feature set
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
           *    64-bit World wide name
           *    URG for READ_STREAM[_DMA]_EXT
           *    URG for WRITE_STREAM[_DMA]_EXT
           *    IDLE_IMMEDIATE with UNLOAD
           *    WRITE_UNCORRECTABLE_EXT command
           *    {READ,WRITE}_DMA_EXT_GPL commands
           *    Segmented DOWNLOAD_MICROCODE
                unknown 119[7]
           *    Gen1 signaling speed (1.5Gb/s)
           *    Gen2 signaling speed (3.0Gb/s)
           *    Gen3 signaling speed (6.0Gb/s)
           *    Native Command Queueing (NCQ)
           *    Host-initiated interface power management
           *    Phy event counters
           *    Idle-Unload when NCQ is active
           *    NCQ priority information
           *    READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
           *    DMA Setup Auto-Activate optimization
                Device-initiated interface power management
           *    Software settings preservation
           *    SMART Command Transport (SCT) feature set
           *    SCT Write Same (AC2)
           *    SCT Error Recovery Control (AC3)
           *    SCT Features Control (AC4)
           *    SCT Data Tables (AC5)
                unknown 206[7]
                unknown 206[12] (vendor specific)
                unknown 206[13] (vendor specific)
                unknown 206[14] (vendor specific)
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
                frozen
        not     expired: security count
                supported: enhanced erase
        426min for SECURITY ERASE UNIT. 426min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 50014ee260f9c90e
        NAA             : 5
        IEEE OUI        : 0014ee
        Unique ID       : 260f9c90e
/dev/sdb:

ATA device, with non-removable media
        Model Number:       ST4000DM000-1F2168
        Serial Number:      W300596N
        Firmware Revision:  CC52
        Transport:          Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
        Used: unknown (minor revision code 0x0029)
        Supported: 8 7 6 5
        Likely used: 8
Configuration:
        Logical         max     current
        cylinders       16383   16383
        heads           16      16
        sectors/track   63      63
        --
        CHS current addressable sectors:    16514064
        LBA    user addressable sectors:   268435455
        LBA48  user addressable sectors:  7814037168
        Logical  Sector size:                   512 bytes
        Physical Sector size:                  4096 bytes
        Logical Sector-0 offset:                  0 bytes
        device size with M = 1024*1024:     3815447 MBytes
        device size with M = 1000*1000:     4000787 MBytes (4000 GB)
        cache/buffer size  = unknown
        Form Factor: 3.5 inch
        Nominal Media Rotation Rate: 5900
Capabilities:
        LBA, IORDY(can be disabled)
        Queue depth: 32
        Standby timer values: spec'd by Standard, no device specific minimum
        R/W multiple sector transfer: Max = 16  Current = 16
        Advanced power management level: 128
        Recommended acoustic management value: 208, current value: 0
        DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
             Cycle time: min=120ns recommended=120ns
        PIO: pio0 pio1 pio2 pio3 pio4
             Cycle time: no flow control=120ns  IORDY flow control=120ns
Commands/features:
        Enabled Supported:
           *    SMART feature set
                Security Mode feature set
           *    Power Management feature set
           *    Write cache
           *    Look-ahead
           *    Host Protected Area feature set
           *    WRITE_BUFFER command
           *    READ_BUFFER command
           *    DOWNLOAD_MICROCODE
           *    Advanced Power Management feature set
                Power-Up In Standby feature set
           *    SET_FEATURES required to spinup after power up
                SET_MAX security extension
           *    48-bit Address feature set
           *    Device Configuration Overlay feature set
           *    Mandatory FLUSH_CACHE
           *    FLUSH_CACHE_EXT
           *    SMART error logging
           *    SMART self-test
           *    General Purpose Logging feature set
           *    WRITE_{DMA|MULTIPLE}_FUA_EXT
           *    64-bit World wide name
                Write-Read-Verify feature set
           *    WRITE_UNCORRECTABLE_EXT command
           *    {READ,WRITE}_DMA_EXT_GPL commands
           *    Segmented DOWNLOAD_MICROCODE
           *    Gen1 signaling speed (1.5Gb/s)
           *    Gen2 signaling speed (3.0Gb/s)
           *    Gen3 signaling speed (6.0Gb/s)
           *    Native Command Queueing (NCQ)
           *    Phy event counters
           *    READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
           *    DMA Setup Auto-Activate optimization
                Device-initiated interface power management
           *    Software settings preservation
           *    SMART Command Transport (SCT) feature set
           *    SCT Write Same (AC2)
                unknown 206[7]
                unknown 206[12] (vendor specific)
Security:
        Master password revision code = 65534
                supported
        not     enabled
        not     locked
                frozen
        not     expired: security count
                supported: enhanced erase
        500min for SECURITY ERASE UNIT. 500min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5000c500606b3c34
        NAA             : 5
        IEEE OUI        : 000c50
        Unique ID       : 0606b3c34
Checksum: correct
sudo smartctl -H /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.8.13-1-ARCH] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

[tremor@Arc-Store-ZYX ~]$ sudo smartctl -a /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.8.13-1-ARCH] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red Pro
Device Model:     WDC WD4001FFSX-68JNUN0
Serial Number:    WD-WCC5D1ZCNKK4
LU WWN Device Id: 5 0014ee 260f9c90e
Firmware Version: 81.00A81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Dec 28 23:18:57 2016 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (40080) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 434) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   157   143   021    Pre-fail  Always       -       11150
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       97
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       11561
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       97
 16 Unknown_Attribute       0x0022   135   065   000    Old_age   Always       -       2377984690560
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       43
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2395
194 Temperature_Celsius     0x0022   113   107   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
sudo smartctl -a /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.8.13-1-ARCH] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red Pro
Device Model:     WDC WD4001FFSX-68JNUN0
Serial Number:    WD-WCC5D1ZCNKK4
LU WWN Device Id: 5 0014ee 260f9c90e
Firmware Version: 81.00A81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Dec 28 23:18:57 2016 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (40080) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 434) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   157   143   021    Pre-fail  Always       -       11150
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       97
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       11561
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       97
 16 Unknown_Attribute       0x0022   135   065   000    Old_age   Always       -       2377984690560
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       43
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       2395
194 Temperature_Celsius     0x0022   113   107   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
sudo smartctl -a /dev/sdb
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.8.13-1-ARCH] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Desktop HDD.15
Device Model:     ST4000DM000-1F2168
Serial Number:    W300596N
LU WWN Device Id: 5 000c50 0606b3c34
Firmware Version: CC52
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5900 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS T13/1699-D revision 4
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed Dec 28 23:19:59 2016 CST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  612) seconds.
Offline data collection
capabilities:                    (0x73) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        No Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        ( 517) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x1085) SCT Status supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   105   099   006    Pre-fail  Always       -       9073136
  3 Spin_Up_Time            0x0003   092   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       225
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   078   060   030    Pre-fail  Always       -       4362377716
  9 Power_On_Hours          0x0032   072   072   000    Old_age   Always       -       24781
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       176
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
184 End-to-End_Error        0x0032   100   100   099    Old_age   Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   099   000    Old_age   Always       -       8 8 8
189 High_Fly_Writes         0x003a   096   096   000    Old_age   Always       -       4
190 Airflow_Temperature_Cel 0x0022   064   059   045    Old_age   Always       -       36 (Min/Max 24/37)
191 G-Sense_Error_Rate      0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       142
193 Load_Cycle_Count        0x0032   097   097   000    Old_age   Always       -       6303
194 Temperature_Celsius     0x0022   036   041   000    Old_age   Always       -       36 (0 17 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       24032h+16m+55.946s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       60266006066
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       124352147735

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     11887         -
# 2  Extended offline    Completed without error       00%     11879         -
# 3  Extended offline    Completed without error       00%     11871         -
# 4  Extended offline    Completed without error       00%     11169         -
# 5  Extended offline    Completed without error       00%     11160         -
# 6  Extended offline    Completed without error       00%     11152         -
# 7  Extended offline    Completed without error       00%     10425         -
# 8  Extended offline    Completed without error       00%     10416         -
# 9  Extended offline    Completed without error       00%     10407         -
#10  Extended offline    Completed without error       00%      9707         -
#11  Extended offline    Completed without error       00%      9698         -
#12  Extended offline    Completed without error       00%      9689         -
#13  Extended offline    Interrupted (host reset)      00%      6838         -
#14  Extended offline    Interrupted (host reset)      80%      6837         -
#15  Extended offline    Completed without error       00%      6835         -
#16  Extended offline    Completed without error       00%      6826         -
#17  Extended offline    Completed without error       00%      6373         -
#18  Extended offline    Completed without error       00%      6349         -
#19  Extended offline    Completed without error       00%      6207         -
#20  Extended offline    Completed without error       00%      6183         -
#21  Extended offline    Completed without error       00%      6039         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

... And now I see both have some pre-fail conditions on their SMART readings.

Alrighty then. ._. Glad you asked that sgtawesomesauce.

7

So those drives are getting a bit old. It's definitely possible one or more of them is about to fail. Try doing this to a file in each drive and report the speed results. 

dd if=/dev/urandom of=/mnt/sdX/file.img bs=4M count=1024

and see what that results in. This should give you the potential that the specific disk formatted the way it is can do. That should give you a 4GB file.

8

Didn't Wendell specifically mention that writing from /dev/urandom can be slow due to the work required to generate random data? He mentioned it in his Tek Syndicate RAID videos when he was using scripts to test how hardware RAID, mdadm, and btrfs/zfs handle one drive being corrupt while the other is not if they are in RAID 1. I only remember because I just watched it recently.

I think he mentioned it averages 15 MB/s or something.

I have an SSD in the system, so I may try that.

9

I averaged 245mb/s to my 10 year old SSD about 5 minutes ago. Tested it just to be sure. You could also do /dev/zero, urandom is just more fun.

1 Like
10

Hmm, I get 40 MB/s off the WD Red Pro and 15 MB/s off the Seagate using /dev/urandom.

Guess the Seagate is dying. And that's the one I'm copying the data to.

Oh well, I'll let it complete just because "two copies".

I'll have to scrounge up the money for a new drive soon. Thanks!

11

I would definitely get a new drive. Also, I wouldn't continue to use it. Sometimes when they die, they think they're returning good data, but they're not. Just trash it when you can.

1 Like
12

Yes, I would if I had a spare at this moment.

Fortunately, it's using BtrFS. And while I know that doesn't make it bullet proof, I am happier knowing there are checksums to be... well checked to verify if it's returning bad data or not. Also, ECC RAM, so less of a danger of scrubs doing bad things.

I'll probably let it finish copying for the above reason then power down the WD Red Pro until I get a spare to move to a safe drive.

I view the Seagate as a burner drive. If it dies, it dies.

Most of this is just backups of nothing and media. I wouldn't be too terribly inconvenienced if it was lost, but it would be annoying.

13

True. You're probably in a pretty safe situation.

1 Like
14

Just to be clear: why do you conclude the drive is dying?

1 Like
15

The SMART Data states both the drives are in pre-fail condition.

For the WD Red Pro:

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail
3 Spin_Up_Time 0x0027 157 143 021 Pre-fail
4 Start_Stop_Count 0x0032 100 100 000 Old_age
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail

For the Seagate:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE
1 Raw_Read_Error_Rate 0x000f 105 099 006 Pre-fail
3 Spin_Up_Time 0x0003 092 091 000 Pre-fail
4 Start_Stop_Count 0x0032 100 100 020 Old_age
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail
7 Seek_Error_Rate 0x000f 078 060 030 Pre-fail
9 Power_On_Hours 0x0032 072 072 000 Old_age
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail

I know for a fact the Seagate should transfer at >20MB/s and the WD Red Pro should transfer at >40MB/s.

I will probably RMA them. They are well within warranty.

16

Ok. So either I have gotten SMART all wrong (admittently it has been a while) or Pre-fail is not a status at all. As long as the actual values are above the threshold for Pre-fail type parameters, you should be ok.

Edit: so looking at https://en.wikipedia.org/wiki/S.M.A.R.T.: Each attribute has a raw value, whose meaning is entirely up to the drive manufacturer (but often corresponds to counts or a physical unit, such as degrees Celsius or seconds), a normalized value, which ranges from 1 to 253 (with 1 representing the worst case and 253 representing the best) and a worst value, which represents the lowest recorded normalized value.

I think your performance is due to the fact you are using a single rsync thread and random read/write patterns on single drives, one having 5400 rpm. You could rsync 2 or more top level directories separately to see if that speeds things up. You could also dd the raw device into /dev/null and see how much MB the drive can deliver at most.

17

If it's due to the 5400RPM drive and a single rsync thread, why does this command:

dd if=/dev/urandom of=/dev/sdX bs=64K count=20000

Transfer at 15 MB/s to my Seagate and 40 MB/s to my WD Red Pro?

That implies to me it's the drives that are the bottleneck. Though I get near constant 19-20 MB/s when using Rsync between the two drives, so I"m not entirely sure on that either.

I'll try multiple rsync threads then.

1 Like
18

I'm not claiming your drives are perfect, I'm just trying to address SMART seems ok so you might want to consider other options first before taking off on a warranty replacement journey.

Just on top of my head:

  • what's the story with this dd coomand? I can't imagine you are blanking your drives all the time. Do you use it on a second partition? Or on the full drive? I would expect higher rates for both discs provided you bypass the filesystem completely.
  • I have had abysmal performance using /dev/urandom. Just use /dev/zero for the source to avoid any misbehavior. Should be the same for both drives though.
  • Why limit the dd blocksize to 64kb? Use 4M to make sure you are actually saturate the interface. I'm not against it, I just don't understand why.

If the drive can't deliver more burst speeds using a plain dd input from /dev/zero outputting to the physical device using 4M block sizes, all other actions seem irrelevant to improve it.

Your drives report a logical sector size of 512 bytes, but a physical one of 4k. This is important. Make sure your partition is properly aligned (parted will take care of it and allow you to verify alignment, it basically means the partition should start at a 4k boundary.) Next to that, make sure your ntfs filesystem is aware of this. I'm not sure whether setting the sector size has any effect on this, but ntfsinfo will show the parameters of the current filesystem.

19

So would I, and that's the issue. I should point out that I wrote that command wrong. I was not outputting directly to the device, but to it's mount point like so: of=/mnt/media/test.img. Essentially just creating a file and in that case, actually using the File System.

I have an SSD in this system. Outputting /dev/urandom to a file on that SSD with the same settings (bs=64K count=20000) gets me just at 40MB/s. So I imagine my WD Red Pro could go faster since that was also it's limit, but my Seagate is seemingly crippled at 15 MB/s (even though an rsync between the two HDDs constantly pushes 19-20 MB/s).

So my guess is that my /dev/urandom's speed limit is 40 MB/s. Using /dev/zero doesn't up that 40 MB/s limit on my WD Red Pro though, so maybe that is also it's limit? Not sure.

Using 4M instead of 64K returns a slightly higher average transfer rate to the SSD (40.4 MB/s vs 40.2MB/s) when using /dev/urandom. When using /dev/zero, it's 650 MB/s (my SSD can't handle that, so I imagine some compression is happening along the way), so /dev/urandom's limit is 40 MB/s and my system isn't handicapped to only 40 MB/s transfers. It's also 650 MB/s when using bs 64K.

Seagate:

WD Red Pro:

The above were tests done with NTFS drives on Windows 10. Guess it's my system?

20

Your rsync command is doing single thread random IO from one disk to another. So on the one hand you are reading random IO from the WD Red using btrfs, and on the other hand you are writing random data to the Seagate NTFS drive that just has 5900 rpm. It is the worst case scenario especially when having a lot of very small files although I must admit this performance is one of the worst I have ever seen.

Do you have a lot of snapshots on the btrfs volume, how much % of it is used?

It's difficult to call. But having the issue on 2 different drives using 2 different operating systems and only with single threaded small random IO? I don't think it's the hardware (yet). Running multiple threads and bigger IO's would be an interesting use case for CDM. That should make a noticeable difference.

BTW do you see any errors in dmesg output?

1 Like