So a lot of cool hardware uses 520 byte sectors instead of 512 byte sectors (or 4* sector size in the case of SSDs).
There is a lot of misinformation out there. Prettymuch anything since 2012 that supports 520 byte sectors will support 512 byte sectors if you know the right magic commands.
It’s possible to acquire SAS SSDs that are End Of Life (lol not really) for fairly cheap. 2TB SAS SSDs go for $200-$350 on ebay and less in bulk and direct surplus.
85% of the time enterprises are replacing SSDs when they are full because LSI, CISCO, Dell, etc have the most amazingly shit raid controllers that DO NOT pass through or even manage trim properly. What happens? The array performance tanks and monitoring metrics show massive performance loss. IT Admins assume the drive has reached end-of-life. What’s actually happening is the drives are hitting their first complete write cycle and, because no blocks are freed (because no trim) performance tanks.
Some drives, such as sandisk, have SOME firmware measures to deal with this a little less braindead, but most dont.
The end result is that you, the deal hunter, can get a great deal on used SAS SSDs.
These are not sata and will not work on motherboard sata headers. I’m using an LSI2308 controller for this how-to.
So, when you boot up you run lsblk, and find you don’t have anything.
You modprobe mpt3sas and see that “Scsi Generic” is detected, but you get some errors such as:
[ 2421.515607] mpt2sas_cm0: _scsih_sas_broadcast_primitive_event: enter: phy number(0), width(1)
[ 2421.515614] sd 6:0:0:0: device_block, handle(0x0009)
[ 2421.515618] sd 6:0:1:0: device_block, handle(0x000a)
[ 2421.515621] sd 6:0:2:0: device_block, handle(0x000b)
[ 2421.515672] sd 6:0:0:0: device_unblock and setting to running, handle(0x0009)
[ 2421.515675] sd 6:0:1:0: device_unblock and setting to running, handle(0x000a)
[ 2421.515677] sd 6:0:2:0: device_unblock and setting to running, handle(0x000b)
[ 2421.760496] .
[ 2422.270567] scsi 6:0:3:0: Direct-Access SanDisk DOPE1920S5xnNMRI 3P01 PQ: 0 ANSI: 6
[ 2422.270580] scsi 6:0:3:0: SSP: handle(0x000c), sas_addr(0x50011731004624bd), phy(0), device_name(0x50011731004624bc)
[ 2422.270582] scsi 6:0:3:0: enclosure logical id (0x500605b009b87b50), slot(1)
[ 2422.270706] scsi 6:0:3:0: Power-on or device reset occurred
[ 2422.271546] sd 6:0:3:0: Attached scsi generic sg4 type 0
[ 2422.271744] sd 6:0:3:0: [sde] Spinning up disk…
What do?
sg_format to the rescue!
root@ubuntu:~# sg_format /dev/sg2
SanDisk DOPE1920S5xnNMRI 3P01 peripheral_type: disk [0x0]
<< supports protection information>>
Unit serial number: 00028FA6
LU name: 50011731004624c0
Mode Sense (block descriptor) data, prior to changes:
Number of blocks=3750748848 [0xdf8fe2b0]
Block size=520 [0x208]
Read Capacity (10) results:
Number of logical blocks=3750748848
Logical block size=520 bytes
No changes made. To format use '--format'. To resize use '--resize'
So we can use this to format to 512 bytes, right? Usually… but…
root@ubuntu:~# sg_format -v --format --size=512 /dev/sg2
SanDisk DOPE1920S5xnNMRI 3P01 peripheral_type: disk [0x0]
PROTECT=1
<< supports protection information>>
Unit serial number: 00028FA6
LU name: 50011731004624c0
mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00
Mode Sense (block descriptor) data, prior to changes:
Number of blocks=3750748848 [0xdf8fe2b0]
Block size=520 [0x208]
mode select (10) cdb: 55 11 00 00 00 00 00 00 1a 00
mode select (10):
Fixed format, current; Sense key: Illegal Request
Additional sense: Parameter list length error
Sense Key Specific: Error in Data parameters: byte 0
MODE SELECT command: Illegal request sense key, apart from Invalid opcode
Oh noooo it wont format will it require something arcane? Nah. This is just the drives way of telling you that it is destructive. Different kinds of drives require different massage here. You might have to do something like
dd if=/dev/zero of=/dev/sg??? bs=104857600
to zero the device. Or just us sg_format to zero the drive with 520 byte sectors:
root@ubuntu:~# sg_format -v --format --size=520 /dev/sg2
SanDisk DOPE1920S5xnNMRI 3P01 peripheral_type: disk [0x0]
PROTECT=1
<< supports protection information>>
Unit serial number: 00028FA6
LU name: 50011731004624c0
mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00
Mode Sense (block descriptor) data, prior to changes:
Number of blocks=3750748848 [0xdf8fe2b0]
Block size=520 [0x208]
A FORMAT UNIT will commence in 15 seconds
ALL data on /dev/sg2 will be DESTROYED
Press control-C to abort
A FORMAT UNIT will commence in 10 seconds
ALL data on /dev/sg2 will be DESTROYED
Press control-C to abort
A FORMAT UNIT will commence in 5 seconds
ALL data on /dev/sg2 will be DESTROYED
Press control-C to abort
format unit cdb: 04 18 00 00 00 00
Format unit has started
Format in progress, 40.71% done
It’s not unusual for the drive to get SUPER hot during this erasure, or for this erasure to take 10 minutes to 10 hours. Depending on several factors. And firmware versions.
While that’s going it’d be a good idea to see if your drive has a firmware update from the manufacturer. Generally, it’s a great idea to install those. But I’m not going to go into that here.
Now, the magic here is that the drive KNOWS it was just formatted and is empty. NOW we can change the sector size! In older drives (pre 2012, and rarely) it might have been necessary to use a different firmware that supported 520 byte sectors instead of 512 bytes. Generally that’s not a thing anymore, though. Aint nobody got time for that. Server POST times are long enough as it is.
Now we can re-issue our format command. Note that if you prod the drive or write even one byte to it, it will be locked for formatting once again and you will be unable to format it to 512 bytes.
Reissue the command, with 512 bytes instead of 520, immediately (and it works! no more illegal command sense!)
oot@ubuntu:~# sg_format -v --format --size=512 /dev/sg2
SanDisk DOPE1920S5xnNMRI 3P01 peripheral_type: disk [0x0]
PROTECT=1
<< supports protection information>>
Unit serial number: 00028FA6
LU name: 50011731004624c0
mode sense (10) cdb: 5a 00 01 00 00 00 00 00 fc 00
Mode Sense (block descriptor) data, prior to changes:
Number of blocks=3750748848 [0xdf8fe2b0]
Block size=512 [0x200]
A FORMAT UNIT will commence in 15 seconds
ALL data on /dev/sg2 will be DESTROYED
Press control-C to abort
TADA! You now have freshly formatted awesome 2tb SAS SSDs. A bunch of them. For science. And ZFS arrays.
[ 224.108169] Fusion MPT base driver 3.04.20
[ 224.108171] Copyright © 1999-2008 LSI Corporation
[ 229.602914] Fusion MPT SAS Host driver 3.04.20
[ 242.538999] mpt3sas 0000:07:00.0: invalid short VPD tag 00 at offset 1
[ 277.590329] sd 6:0:0:0: [sda] Unsupported sector size 520.
[ 280.444460] sd 6:0:0:0: [sda] Unsupported sector size 520.
[ 887.821141] sd 6:0:0:0: [sda] 3750748848 512-byte logical blocks: (1.92 TB/1.75 TiB)
[ 887.821144] sd 6:0:0:0: [sda] 4096-byte physical blocks
[ 887.821809] sda: detected capacity change from 0 to 1920383410176
Bonus Round
How much life do these have left? How much has been written to them?
try these commands:
root@ubuntu:~# smartctl --all /dev/sda
smartctl 6.6 2016-05-31 r4324 [x86_64-linux-4.15.0-20-generic] (local build)
Copyright © 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Vendor: SanDisk
Product: DOPE1920S5xnNMRI
Revision: 3P01
Compliance: SPC-4
User Capacity: 1,920,383,410,176 bytes [1.92 TB]
Logical block size: 512 bytes
Physical block size: 4096 bytes
LU is resource provisioned, LBPRZ=1
Rotation Rate: Solid State Device
Form Factor: 2.5 inches
Logical Unit id: 0x5001173100462502
Serial number: 00018FB8
Device type: disk
Transport protocol: SAS (SPL-3)
Local Time is: Wed Sep 26 00:17:24 2018 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
Temperature Warning: Enabled
=== START OF READ SMART DATA SECTION ===
SMART Health Status: OK
Percentage used endurance indicator: 0%
Current Drive Temperature: 32 C
Drive Trip Temperature: 63 C
Manufactured in week 44 of year 2014
Specified cycle count over device lifetime: 0
Accumulated start-stop cycles: 93
defect list format 6 unknown
Elements in grown defect list: 0
Error counter log:
Errors Corrected by Total Correction Gigabytes Total
ECC rereads/ errors algorithm processed uncorrected
fast | delayed rewrites corrected invocations [10^9 bytes] errors
read: 0 0 0 0 0 4725.548 0
write: 0 0 0 0 0 **2204.632** 0
Non-medium error count: 0
No self-tests have been logged
So this tells us this MLC drive has had ONLY 2.2 tb of data written to it. That is highly likely these were chucked by someone who wanted to fix the “suddenly these SSDs are slow” problem, which is a lack of trim support from whatever these were running in before.
E.g. These drives are basically brand new. These flash cells were designed for Petabytes of data to be written to them. So I’d say LULZ.
ps… if smartmontools doesn’t do it, you might be able to cat /proc/diskstat but… I kinda doubt it. You’ll have to man that to know which field is bytes (sectors? in a lot of cases) written. Oh, and for doing all this work you should always use an HBA and NOT a raid controller. Why no raid controllers? Because raid controllers are awful garbage generally.