Return to Level1Techs.com

How to reformat 520 byte drives to 512 bytes (usually)

So a lot of cool hardware uses 520 byte sectors instead of 512 byte sectors (or 4* sector size in the case of SSDs).

There is a lot of misinformation out there. Prettymuch anything since 2012 that supports 520 byte sectors will support 512 byte sectors if you know the right magic commands.

It’s possible to acquire SAS SSDs that are End Of Life (lol not really) for fairly cheap. 2TB SAS SSDs go for $200-$350 on ebay and less in bulk and direct surplus.

85% of the time enterprises are replacing SSDs when they are full because LSI, CISCO, Dell, etc have the most amazingly shit raid controllers that DO NOT pass through or even manage trim properly. What happens? The array performance tanks and monitoring metrics show massive performance loss. IT Admins assume the drive has reached end-of-life. What’s actually happening is the drives are hitting their first complete write cycle and, because no blocks are freed (because no trim) performance tanks.

Some drives, such as sandisk, have SOME firmware measures to deal with this a little less braindead, but most dont.

The end result is that you, the deal hunter, can get a great deal on used SAS SSDs.

These are not sata and will not work on motherboard sata headers. I’m using an LSI2308 controller for this how-to.

So, when you boot up you run lsblk, and find you don’t have anything.

You modprobe mpt3sas and see that “Scsi Generic” is detected, but you get some errors such as:

[ 2421.515607] mpt2sas_cm0: _scsih_sas_broadcast_primitive_event: enter: phy number(0), width(1)

[ 2421.515614] sd 6:0:0:0: device_block, handle(0x0009)
[ 2421.515618] sd 6:0:1:0: device_block, handle(0x000a)
[ 2421.515621] sd 6:0:2:0: device_block, handle(0x000b)
[ 2421.515672] sd 6:0:0:0: device_unblock and setting to running, handle(0x0009)
[ 2421.515675] sd 6:0:1:0: device_unblock and setting to running, handle(0x000a)
[ 2421.515677] sd 6:0:2:0: device_unblock and setting to running, handle(0x000b)
[ 2421.760496] .
[ 2422.270567] scsi 6:0:3:0: Direct-Access SanDisk DOPE1920S5xnNMRI 3P01 PQ: 0 ANSI: 6
[ 2422.270580] scsi 6:0:3:0: SSP: handle(0x000c), sas_addr(0x50011731004624bd), phy(0), device_name(0x50011731004624bc)
[ 2422.270582] scsi 6:0:3:0: enclosure logical id (0x500605b009b87b50), slot(1)
[ 2422.270706] scsi 6:0:3:0: Power-on or device reset occurred
[ 2422.271546] sd 6:0:3:0: Attached scsi generic sg4 type 0
[ 2422.271744] sd 6:0:3:0: [sde] Spinning up disk…

What do?

sg_format to the rescue!

[email protected]:~# sg_format /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=520 [0x208]
Read Capacity (10) results:
   Number of logical blocks=3750748848
   Logical block size=520 bytes
No changes made. To format use '--format'. To resize use '--resize'

So we can use this to format to 512 bytes, right? Usually… but…

[email protected]:~# sg_format -v --format --size=512 /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=520 [0x208]
    mode select (10) cdb: 55 11 00 00 00 00 00 00 1a 00 
mode select (10):
Fixed format, current; Sense key: Illegal Request
Additional sense: Parameter list length error
  Sense Key Specific: Error in Data parameters: byte 0
MODE SELECT command: Illegal request sense key, apart from Invalid opcode

Oh noooo it wont format will it require something arcane? Nah. This is just the drives way of telling you that it is destructive. Different kinds of drives require different massage here. You might have to do something like

dd if=/dev/zero of=/dev/sg??? bs=104857600

to zero the device. Or just us sg_format to zero the drive with 520 byte sectors:

[email protected]:~# sg_format -v --format --size=520 /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=520 [0x208]

A FORMAT UNIT will commence in 15 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort

A FORMAT UNIT will commence in 10 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort

A FORMAT UNIT will commence in 5 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort
    format unit cdb: 04 18 00 00 00 00 

Format unit has started
Format in progress, 40.71% done

It’s not unusual for the drive to get SUPER hot during this erasure, or for this erasure to take 10 minutes to 10 hours. Depending on several factors. And firmware versions.

While that’s going it’d be a good idea to see if your drive has a firmware update from the manufacturer. Generally, it’s a great idea to install those. But I’m not going to go into that here.

Now, the magic here is that the drive KNOWS it was just formatted and is empty. NOW we can change the sector size! In older drives (pre 2012, and rarely) it might have been necessary to use a different firmware that supported 520 byte sectors instead of 512 bytes. Generally that’s not a thing anymore, though. Aint nobody got time for that. Server POST times are long enough as it is.

Now we can re-issue our format command. Note that if you prod the drive or write even one byte to it, it will be locked for formatting once again and you will be unable to format it to 512 bytes.

Reissue the command, with 512 bytes instead of 520, immediately (and it works! no more illegal command sense!)

[email protected]:~# sg_format -v --format --size=512 /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=512 [0x200]

A FORMAT UNIT will commence in 15 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort

TADA! You now have freshly formatted awesome 2tb SAS SSDs. A bunch of them. For science. And ZFS arrays.

[ 224.108169] Fusion MPT base driver 3.04.20
[ 224.108171] Copyright © 1999-2008 LSI Corporation
[ 229.602914] Fusion MPT SAS Host driver 3.04.20
[ 242.538999] mpt3sas 0000:07:00.0: invalid short VPD tag 00 at offset 1
[ 277.590329] sd 6:0:0:0: [sda] Unsupported sector size 520.
[ 280.444460] sd 6:0:0:0: [sda] Unsupported sector size 520.
[ 887.821141] sd 6:0:0:0: [sda] 3750748848 512-byte logical blocks: (1.92 TB/1.75 TiB)
[ 887.821144] sd 6:0:0:0: [sda] 4096-byte physical blocks
[ 887.821809] sda: detected capacity change from 0 to 1920383410176

Bonus Round

How much life do these have left? How much has been written to them?

try these commands:
[email protected]:~# smartctl --all /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-20-generic] (local build)
Copyright © 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SanDisk
Product:              DOPE1920S5xnNMRI
Revision:             3P01
Compliance:           SPC-4
User Capacity:        1,920,383,410,176 bytes [1.92 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate:        Solid State Device
Form Factor:          2.5 inches
Logical Unit id:      0x5001173100462502
Serial number:        00018FB8
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Sep 26 00:17:24 2018 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Percentage used endurance indicator: 0%
Current Drive Temperature:     32 C
Drive Trip Temperature:        63 C

Manufactured in week 44 of year 2014
Specified cycle count over device lifetime:  0
Accumulated start-stop cycles:  93
defect list format 6 unknown
Elements in grown defect list: 0

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0          0       4725.548           0
write:         0        0         0         0          0      **2204.632**           0

Non-medium error count:        0

No self-tests have been logged

So this tells us this MLC drive has had ONLY 2.2 tb of data written to it. That is highly likely these were chucked by someone who wanted to fix the “suddenly these SSDs are slow” problem, which is a lack of trim support from whatever these were running in before.

E.g. These drives are basically brand new. These flash cells were designed for Petabytes of data to be written to them. So I’d say LULZ.

ps… if smartmontools doesn’t do it, you might be able to cat /proc/diskstat but… I kinda doubt it. You’ll have to man that to know which field is bytes (sectors? in a lot of cases) written. Oh, and for doing all this work you should always use an HBA and NOT a raid controller. Why no raid controllers? Because raid controllers are awful garbage generally.

:smiley:

12 Likes

One week

8tb of flash for “oh no it’s broken” prices is a nice thing.

5 Likes

Is there a way to stream line this for say 72 drives :slight_smile:

Is there any way to just keep using it with 520 byte blocks?
Is 520 just unsupported by Linux? Would a BSD or Solaris/Illumos OS handle it properly instead?

1 Like

Bash script? :slight_smile:

1 Like

thanks for the the direction wendell never used linux before time to start learning :slight_smile:

Hi wendell,

Thanks for the awesome guide.
I did the same as you and the drives succesfully formated with sgutils to 512 byte sector sized drives. The only issue I am facing now is that I can’t format de disks with a filesystem… In windows disk magagement I am able to initialize as a GPT drive. But creating a simplevolume will not work…

I am running a dell r710 with a h200 in IT mode and as a host OS i am running ESXI 6.7 .
I am passing that HBA through to the Win VM.
The drives I have are Seagate Constellations 3TB drives.

Do you have any idea what the cause can be ?
I have also tried to create a datastore directly under ESXI but that also does not work.

Errors from trying to do it with parted under ubuntu 18.04:

Blockquote
Jan 28 10:52:05 ubuntubak systemd-udevd[2863]: Process ‘/sbin/mdadm -If sda1 --path pci-0000:03:00.0-sas-phy3-lun-0’ failed with exit code 1.
Jan 28 10:52:05 ubuntubak systemd-udevd[2864]: Process ‘/sbin/mdadm -If sda2 --path pci-0000:03:00.0-sas-phy3-lun-0’ failed with exit code 1.
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695085] sd 16:0:4:0: [sde] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695089] sd 16:0:4:0: [sde] tag#1 Sense Key : Hardware Error [current] [descriptor]
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695092] sd 16:0:4:0: [sde] tag#1 <>ASC=0x81 ASCQ=0x0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695095] sd 16:0:4:0: [sde] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 00 00 00 00 00 00 18 00 00
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695097] print_req_error: critical target error, dev sde, sector 0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695651] buffer_io_error: 6 callbacks suppressed
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695652] Buffer I/O error on dev sde, logical block 0, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.696177] Buffer I/O error on dev sde, logical block 1, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.696670] Buffer I/O error on dev sde, logical block 2, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697153] sd 16:0:4:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697156] sd 16:0:4:0: [sde] tag#0 Sense Key : Hardware Error [current] [descriptor]
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697159] sd 16:0:4:0: [sde] tag#0 <>ASC=0x81 ASCQ=0x0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697161] sd 16:0:4:0: [sde] tag#0 CDB: Write(16) 8a 00 00 00 00 01 5d 50 a3 98 00 00 00 18 00 00
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697162] print_req_error: critical target error, dev sde, sector 5860533144
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697786] Buffer I/O error on dev sde, logical block 732566643, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.698365] Buffer I/O error on dev sde, logical block 732566644, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.698888] Buffer I/O error on dev sde, logical block 732566645, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872079] sd 16:0:4:0: [sde] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872156] sd 16:0:4:0: [sde] tag#1 Sense Key : Hardware Error [current] [descriptor]
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872165] sd 16:0:4:0: [sde] tag#1 <>ASC=0x81 ASCQ=0x0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872172] sd 16:0:4:0: [sde] tag#1 CDB: Write(16) 8a 00 00 00 00 01 5d 50 a3 88 00 00 00 28 00 00

Thanks,

Mat.

I know I’m reopening an old thread but like Mat, I ran into the same issue and I did not find a solution on the L1 tech forums. Hopefully, this helps someone else. After mountains of reading, I found an argument you can set to format these drives when having those errors. --six is the argument for mode sense to set 6 byte mode sense instead of the default 10-byte mode sense. Netapp uses a 10byte mode but 4 are for checksum so you have to use the 6-byte option. I’m sure others use a similar setup. The final command will look something like this sudo sg_format -v --format --size=512 --six /dev/sdx

I’m currently using a PowerEdge R420 with a Dell HBA (03ddjt) H200e and a Netapp ds 2246 disk shelf.

5 Likes