How to reformat 520 byte drives to 512 bytes (usually)

wendell · September 26, 2018, 12:07am

So a lot of cool hardware uses 520 byte sectors instead of 512 byte sectors (or 4* sector size in the case of SSDs).

There is a lot of misinformation out there. Prettymuch anything since 2012 that supports 520 byte sectors will support 512 byte sectors if you know the right magic commands.

It’s possible to acquire SAS SSDs that are End Of Life (lol not really) for fairly cheap. 2TB SAS SSDs go for $200-$350 on ebay and less in bulk and direct surplus.

85% of the time enterprises are replacing SSDs when they are full because LSI, CISCO, Dell, etc have the most amazingly shit raid controllers that DO NOT pass through or even manage trim properly. What happens? The array performance tanks and monitoring metrics show massive performance loss. IT Admins assume the drive has reached end-of-life. What’s actually happening is the drives are hitting their first complete write cycle and, because no blocks are freed (because no trim) performance tanks.

Some drives, such as sandisk, have SOME firmware measures to deal with this a little less braindead, but most dont.

The end result is that you, the deal hunter, can get a great deal on used SAS SSDs.

These are not sata and will not work on motherboard sata headers. I’m using an LSI2308 controller for this how-to.

So, when you boot up you run lsblk, and find you don’t have anything.

You modprobe mpt3sas and see that “Scsi Generic” is detected, but you get some errors such as:

[ 2421.515607] mpt2sas_cm0: _scsih_sas_broadcast_primitive_event: enter: phy number(0), width(1)

[ 2421.515614] sd 6:0:0:0: device_block, handle(0x0009)
[ 2421.515618] sd 6:0:1:0: device_block, handle(0x000a)
[ 2421.515621] sd 6:0:2:0: device_block, handle(0x000b)
[ 2421.515672] sd 6:0:0:0: device_unblock and setting to running, handle(0x0009)
[ 2421.515675] sd 6:0:1:0: device_unblock and setting to running, handle(0x000a)
[ 2421.515677] sd 6:0:2:0: device_unblock and setting to running, handle(0x000b)
[ 2421.760496] .
[ 2422.270567] scsi 6:0:3:0: Direct-Access SanDisk DOPE1920S5xnNMRI 3P01 PQ: 0 ANSI: 6
[ 2422.270580] scsi 6:0:3:0: SSP: handle(0x000c), sas_addr(0x50011731004624bd), phy(0), device_name(0x50011731004624bc)
[ 2422.270582] scsi 6:0:3:0: enclosure logical id (0x500605b009b87b50), slot(1)
[ 2422.270706] scsi 6:0:3:0: Power-on or device reset occurred
[ 2422.271546] sd 6:0:3:0: Attached scsi generic sg4 type 0
[ 2422.271744] sd 6:0:3:0: [sde] Spinning up disk…

What do?

sg_format to the rescue!

root@ubuntu:~# sg_format /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=520 [0x208]
Read Capacity (10) results:
   Number of logical blocks=3750748848
   Logical block size=520 bytes
No changes made. To format use '--format'. To resize use '--resize'

So we can use this to format to 512 bytes, right? Usually… but…

root@ubuntu:~# sg_format -v --format --size=512 /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=520 [0x208]
    mode select (10) cdb: 55 11 00 00 00 00 00 00 1a 00 
mode select (10):
Fixed format, current; Sense key: Illegal Request
Additional sense: Parameter list length error
  Sense Key Specific: Error in Data parameters: byte 0
MODE SELECT command: Illegal request sense key, apart from Invalid opcode

Oh noooo it wont format will it require something arcane? Nah. This is just the drives way of telling you that it is destructive. Different kinds of drives require different massage here. You might have to do something like

dd if=/dev/zero of=/dev/sg??? bs=104857600

to zero the device. Or just us sg_format to zero the drive with 520 byte sectors:

root@ubuntu:~# sg_format -v --format --size=520 /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=520 [0x208]

A FORMAT UNIT will commence in 15 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort

A FORMAT UNIT will commence in 10 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort

A FORMAT UNIT will commence in 5 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort
    format unit cdb: 04 18 00 00 00 00 

Format unit has started
Format in progress, 40.71% done

It’s not unusual for the drive to get SUPER hot during this erasure, or for this erasure to take 10 minutes to 10 hours. Depending on several factors. And firmware versions.

While that’s going it’d be a good idea to see if your drive has a firmware update from the manufacturer. Generally, it’s a great idea to install those. But I’m not going to go into that here.

Now, the magic here is that the drive KNOWS it was just formatted and is empty. NOW we can change the sector size! In older drives (pre 2012, and rarely) it might have been necessary to use a different firmware that supported 520 byte sectors instead of 512 bytes. Generally that’s not a thing anymore, though. Aint nobody got time for that. Server POST times are long enough as it is.

Now we can re-issue our format command. Note that if you prod the drive or write even one byte to it, it will be locked for formatting once again and you will be unable to format it to 512 bytes.

Reissue the command, with 512 bytes instead of 520, immediately (and it works! no more illegal command sense!)

oot@ubuntu:~# sg_format -v --format --size=512 /dev/sg2
    SanDisk   DOPE1920S5xnNMRI  3P01   peripheral_type: disk [0x0]
      PROTECT=1
      << supports protection information>>
      Unit serial number: 00028FA6
      LU name: 50011731004624c0
    mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00 
Mode Sense (block descriptor) data, prior to changes:
  Number of blocks=3750748848 [0xdf8fe2b0]
  Block size=512 [0x200]

A FORMAT UNIT will commence in 15 seconds
    ALL data on /dev/sg2 will be DESTROYED
        Press control-C to abort

TADA! You now have freshly formatted awesome 2tb SAS SSDs. A bunch of them. For science. And ZFS arrays.

[ 224.108169] Fusion MPT base driver 3.04.20
[ 224.108171] Copyright © 1999-2008 LSI Corporation
[ 229.602914] Fusion MPT SAS Host driver 3.04.20
[ 242.538999] mpt3sas 0000:07:00.0: invalid short VPD tag 00 at offset 1
[ 277.590329] sd 6:0:0:0: [sda] Unsupported sector size 520.
[ 280.444460] sd 6:0:0:0: [sda] Unsupported sector size 520.
[ 887.821141] sd 6:0:0:0: [sda] 3750748848 512-byte logical blocks: (1.92 TB/1.75 TiB)
[ 887.821144] sd 6:0:0:0: [sda] 4096-byte physical blocks
[ 887.821809] sda: detected capacity change from 0 to 1920383410176

Bonus Round

How much life do these have left? How much has been written to them?

try these commands:
root@ubuntu:~# smartctl --all /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-20-generic] (local build)
Copyright © 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Vendor:               SanDisk
Product:              DOPE1920S5xnNMRI
Revision:             3P01
Compliance:           SPC-4
User Capacity:        1,920,383,410,176 bytes [1.92 TB]
Logical block size:   512 bytes
Physical block size:  4096 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate:        Solid State Device
Form Factor:          2.5 inches
Logical Unit id:      0x5001173100462502
Serial number:        00018FB8
Device type:          disk
Transport protocol:   SAS (SPL-3)
Local Time is:        Wed Sep 26 00:17:24 2018 UTC
SMART support is:     Available - device has SMART capability.
SMART support is:     Enabled
Temperature Warning:  Enabled

=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK

Percentage used endurance indicator: 0%
Current Drive Temperature:     32 C
Drive Trip Temperature:        63 C

Manufactured in week 44 of year 2014
Specified cycle count over device lifetime:  0
Accumulated start-stop cycles:  93
defect list format 6 unknown
Elements in grown defect list: 0

Error counter log:
           Errors Corrected by           Total   Correction     Gigabytes    Total
               ECC          rereads/    errors   algorithm      processed    uncorrected
           fast | delayed   rewrites  corrected  invocations   [10^9 bytes]  errors
read:          0        0         0         0          0       4725.548           0
write:         0        0         0         0          0      **2204.632**           0

Non-medium error count:        0

No self-tests have been logged

So this tells us this MLC drive has had ONLY 2.2 tb of data written to it. That is highly likely these were chucked by someone who wanted to fix the “suddenly these SSDs are slow” problem, which is a lack of trim support from whatever these were running in before.

E.g. These drives are basically brand new. These flash cells were designed for Petabytes of data to be written to them. So I’d say LULZ.

ps… if smartmontools doesn’t do it, you might be able to cat /proc/diskstat but… I kinda doubt it. You’ll have to man that to know which field is bytes (sectors? in a lot of cases) written. Oh, and for doing all this work you should always use an HBA and NOT a raid controller. Why no raid controllers? Because raid controllers are awful garbage generally.

Goalkeeper · September 26, 2018, 12:12am

One week

wendell · September 26, 2018, 12:16am

8tb of flash for “oh no it’s broken” prices is a nice thing.

dualbios · January 8, 2019, 12:56pm

Is there a way to stream line this for say 72 drives

orgake · January 8, 2019, 5:42pm

Is there any way to just keep using it with 520 byte blocks?
Is 520 just unsupported by Linux? Would a BSD or Solaris/Illumos OS handle it properly instead?

wendell · January 8, 2019, 5:55pm

Bash script?

dualbios · January 12, 2019, 1:24am

thanks for the the direction wendell never used linux before time to start learning

Matteo_Ventura · January 28, 2019, 10:10am

Hi wendell,

Thanks for the awesome guide.
I did the same as you and the drives succesfully formated with sgutils to 512 byte sector sized drives. The only issue I am facing now is that I can’t format de disks with a filesystem… In windows disk magagement I am able to initialize as a GPT drive. But creating a simplevolume will not work…

I am running a dell r710 with a h200 in IT mode and as a host OS i am running ESXI 6.7 .
I am passing that HBA through to the Win VM.
The drives I have are Seagate Constellations 3TB drives.

Do you have any idea what the cause can be ?
I have also tried to create a datastore directly under ESXI but that also does not work.

Errors from trying to do it with parted under ubuntu 18.04:

Blockquote
Jan 28 10:52:05 ubuntubak systemd-udevd[2863]: Process ‘/sbin/mdadm -If sda1 --path pci-0000:03:00.0-sas-phy3-lun-0’ failed with exit code 1.
Jan 28 10:52:05 ubuntubak systemd-udevd[2864]: Process ‘/sbin/mdadm -If sda2 --path pci-0000:03:00.0-sas-phy3-lun-0’ failed with exit code 1.
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695085] sd 16:0:4:0: [sde] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695089] sd 16:0:4:0: [sde] tag#1 Sense Key : Hardware Error [current] [descriptor]
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695092] sd 16:0:4:0: [sde] tag#1 <>ASC=0x81 ASCQ=0x0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695095] sd 16:0:4:0: [sde] tag#1 CDB: Write(16) 8a 00 00 00 00 00 00 00 00 00 00 00 00 18 00 00
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695097] print_req_error: critical target error, dev sde, sector 0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695651] buffer_io_error: 6 callbacks suppressed
Jan 28 10:52:19 ubuntubak kernel: [ 1682.695652] Buffer I/O error on dev sde, logical block 0, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.696177] Buffer I/O error on dev sde, logical block 1, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.696670] Buffer I/O error on dev sde, logical block 2, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697153] sd 16:0:4:0: [sde] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697156] sd 16:0:4:0: [sde] tag#0 Sense Key : Hardware Error [current] [descriptor]
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697159] sd 16:0:4:0: [sde] tag#0 <>ASC=0x81 ASCQ=0x0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697161] sd 16:0:4:0: [sde] tag#0 CDB: Write(16) 8a 00 00 00 00 01 5d 50 a3 98 00 00 00 18 00 00
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697162] print_req_error: critical target error, dev sde, sector 5860533144
Jan 28 10:52:19 ubuntubak kernel: [ 1682.697786] Buffer I/O error on dev sde, logical block 732566643, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.698365] Buffer I/O error on dev sde, logical block 732566644, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.698888] Buffer I/O error on dev sde, logical block 732566645, lost async page write
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872079] sd 16:0:4:0: [sde] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872156] sd 16:0:4:0: [sde] tag#1 Sense Key : Hardware Error [current] [descriptor]
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872165] sd 16:0:4:0: [sde] tag#1 <>ASC=0x81 ASCQ=0x0
Jan 28 10:52:19 ubuntubak kernel: [ 1682.872172] sd 16:0:4:0: [sde] tag#1 CDB: Write(16) 8a 00 00 00 00 01 5d 50 a3 88 00 00 00 28 00 00

Thanks,

Mat.

CrazyLegsFE · May 31, 2020, 2:13pm

I know I’m reopening an old thread but like Mat, I ran into the same issue and I did not find a solution on the L1 tech forums. Hopefully, this helps someone else. After mountains of reading, I found an argument you can set to format these drives when having those errors. --six is the argument for mode sense to set 6 byte mode sense instead of the default 10-byte mode sense. Netapp uses a 10byte mode but 4 are for checksum so you have to use the 6-byte option. I’m sure others use a similar setup. The final command will look something like this sudo sg_format -v --format --size=512 --six /dev/sdx

I’m currently using a PowerEdge R420 with a Dell HBA (03ddjt) H200e and a Netapp ds 2246 disk shelf.

Rain_One · December 28, 2020, 7:34pm

Recently, I bought one of these SanDisk SSD drives, alongside some platter 8TB drives. They were all converted from 520 byte sectors and came pre-formatted as 512 byte, making them work perfectly on my PERC200 DELL SAS card. No issues there, but they do report as being “7.9TB drives” and the SSD reports itself as being “1.9TB”.

Reading up on the 8TB platter drives, I found out that the size disparity is because, on those drives, 512 byte (logical) sector generates a lot of parity bytes, which makes the size be smaller than 8TB. Re-formatting the drives to 4096 byte sector (logical) aligns them with the physical sectors and solves that issue, as there are less “parity bytes” to keep, providing the drive’s actual size as storage.

Would formatting these Sandisk 2TB drives as 4096 byte sectors also solve the issue of them reporting as 1.9TB? Does it even support it? I see that they also have 4096 sectors physically (both on mine and on your drive diagnostics), so I wondered if it’s the same issue with these unrelated platter drives that show the same issue (less storage than it’s supposed to be). I would try it myself, but my current SAS controller doesn’t support 4kn.

The datasheet from the device (google “sandisk-optimus-eco-sas-ssds-datasheet.pdf”, as I can’t post links) doesn’t mention 4096b support, but I don’t have much experience with SAS drives and controllers, and the device itself reports having 4096 physical blocks, so I’m curious.

Vuurvoske · May 31, 2021, 1:44pm

This saved me alot of frustration. I could finally create a zpool on my 3TB SAS disks. Thnx alot for this guide.

jamjam · June 14, 2021, 8:05pm

Hello everybody

i have ssd 3par sas 6Gb and sas 12Gb with 520 sector , i formatted with sg3_utils to 512 sector successfully but all ssd are very slow in windows hardisk manager when i initialised or formating .

someone have advices for me to help to have ssd more speed ?

tks and soryy for my poor english

wendell · June 14, 2021, 8:11pm

leave them on for many hours as after this format they may be doing a self-trim. Or make sure you controller properly supports trim.

after sending the trim command, after a long while, the drive will be very hot and return to normal.

please make sure your drives have adequate cooling.

Vuurvoske · June 28, 2021, 10:06pm

do you consider this adeqaute cooling?

Vuurvoske · June 28, 2021, 10:21pm

#rewrite bytesize of sectors on SAS disks

#also check: How to reformat 520 byte drives to 512 bytes (usually)

single version:

sg_format -v --format --size=512 /dev/sg2

loop for multiple

for disk in $(ls /dev/da{#first…#last}); do echo “>>> ${disk} <<<” && sg_format -v --format --size=512 ${disk}
; done

replace #first with your first drive number i.e. da0 and #last with the lastnumeber i.e. da12

cossykosmak · August 12, 2021, 5:19pm

So i bought some Samsung EMC SSDs with Sata but they won’t let me format them and i’ve read that Sata doesn’t allow for block size change. Is this true? Or is there any one with the same problem?
The SSDs are MZ3S92000XACP

zzzz_tim · November 10, 2021, 3:52am

Edit: the method tried in this post was not successful. It formats, but does not change the block size.

Just joined up to comment that I tried both the original command, and the ‘–six’ option, and tried formatting at 528 bytes (which my drive came as) then immediately reformatting to 512, but none of these worked (all reported “Illegal request sense key” or similar). For the record, this one is a Seagate Constellation ES2 (DXS2D-H3R0SS/ST33000650SS) 3TB drive.

What appears to have worked in the end was sg_format -F -F -F --size=512 /dev/blah (ie, passing the --format option three times, which the doc says means “FORMAT UNIT command only”, I assume it doesn’t do mode sense / status command first, and I guess that was what was erroring out when the block size didn’t match.)

It is currently in the process of formatting so I am taking that as a success, but I’ll come back and edit this post if it turns out to have failed (if the block size didn’t change).

wendell · November 10, 2021, 3:54am

thanks for posting, denvercoder76

zzzz_tim · November 10, 2021, 3:58am

This was the most comprehensive result I found from searching, so I figured this was the best place to put it.
(Hello, future people tearing your hair out over uncooperative drives.)

wendell · November 10, 2021, 3:59am

fwiw I had a couple of stubborn drives but hot plugging them right before formatting seemed to get their cooperation. Since this original post, and my original 12 drives, four have died.