Help enabling T10-DIX on a SAS disk

hsnyder · March 3, 2024, 6:42pm

Hey everyone. I’ve been on an extended rabbit hole regarding T10-PI (protection information) for SCSI and NVMe devices under Linux. Basically, a set of extensions to the SCSI and NVMe specifications, referred to variously as T10-PI, T10-DIF, and T10-DIX, allow there to be end-to-end data integrity verification through checksums, checked at various parts of the chain connecting the OS to the disk.

If my understanding is correct, in the context of SAS drives,

T10-DIF (data integrity field) is a setup where the HBA performs checksums and transmits them to the disk along with every block. The disk is supposed to verify and store the checksum, and return it, to be checked by the HBA, upon reading that block back.
T10-DIX takes this a step further and allows the OS to compute the checksum. I think the HBA is supposed to verify it, and pass it along to the disk.

Anyways, I’m trying to get this working on real hardware. I have:

A Seagate Exos 20TB SAS drive
An LSI card based on the SAS2008 chip, flashed in IT mode to act as an HBA
An x86_64 Linux host running kernel 6.1.0-17 (debian)

I loaded the mpt3sas driver with the option prot_mask=0x7f, since according to the driver documentation (https://docs.broadcom.com/doc/Linux_Driver-RHEL7-8_SLES12-15_PCIe_P18_0.pdf) this will enable DIF and DIX.

I formatted the disk. Since (I think) DIF works by using larger blocks to store the checksum in addition to the data, at first I tried formatting it with larger sectors (4160 bytes) via sg_format --format --size=4160 --fmtpinfo=2 --pfu=0 --long /dev/sdg. The fmtpinfo flag is supposed to enable Type1 DIF according to the sg_format man page. But after that the detected capacity was zero due to unsupported block size. So I tried again with sg_format --format --size=4096 --fmtpinfo=2 --pfu=0 --long /dev/sdg And now I have a drive that shows up with a usable capacity.

I see a note in dmesg [sdg] Enabling DIF Type 1 protection, which sounds good, and lsscsi -p notes DIF/Type1 beside the drive. But, when I cat the various parameters in /sys/block/sdg/integrity/ it seems like integrity is not actually enabled:

device_is_integrity_capable = 0
format = none

I’ll note that I did all this without a machine restart. I could restart it but the host is in use so it’s inconvenient.

My questions:

I gather DIX is not enabled, but I’m not clear on whether DIF is even working properly. Is it? How do I know?
How would I enable DIX if I wanted to?
Does anyone have any other general advice or knowledge to share?

hsnyder · March 5, 2024, 4:58pm

LiKenun · March 18, 2024, 2:59am

Have you restarted it yet?

hsnyder · March 19, 2024, 5:48pm

No!

I think what’s going on is that Type1 DIF is working, but the HBA doesn’t support DIX. Here’s why I think this:

lsscsi and dmesg say/said that Type 1 DIF is enabled
smartctl -a says

Formatted with type 1 protection
8 bytes of protection information per logical block

I started poking around in the linux source code as well, and it looks like the path /sys/block/sdg/integrity/device_is_integrity_capable is reporting on DIX, not DIF. It’s reporting on the blk_integrity struct, which is only actually registered if DIX is being used, not DIF.

github.com

torvalds/linux/blob/master/drivers/scsi/sd.c#L2443


      
          			  type);
          		sdkp->protection_type = 0;
          		return -ENODEV;
          	}
          
          	sdkp->protection_type = type;
          
          	return 0;
          }
          
          static void sd_config_protection(struct scsi_disk *sdkp)
          {
          	struct scsi_device *sdp = sdkp->device;
          
          	sd_dif_config_host(sdkp);
          
          	if (!sdkp->protection_type)
          		return;
          
          	if (!scsi_host_dif_capable(sdp->host, sdkp->protection_type)) {
          		sd_first_printk(KERN_NOTICE, sdkp,

github.com

torvalds/linux/blob/master/drivers/scsi/sd_dif.c#L27


      
          #include <scsi/scsi_eh.h>
          #include <scsi/scsi_host.h>
          #include <scsi/scsi_ioctl.h>
          #include <scsi/scsicam.h>
          
          #include "sd.h"
          
          /*
           * Configure exchange of protection information between OS and HBA.
           */
          void sd_dif_config_host(struct scsi_disk *sdkp)
          {
          	struct scsi_device *sdp = sdkp->device;
          	struct gendisk *disk = sdkp->disk;
          	u8 type = sdkp->protection_type;
          	struct blk_integrity bi;
          	int dif, dix;
          
          	dif = scsi_host_dif_capable(sdp->host, type);
          	dix = scsi_host_dix_capable(sdp->host, type);

github.com

torvalds/linux/blob/master/block/blk-integrity.c#L297


      
          }
          
          static ssize_t write_generate_show(struct device *dev,
          				   struct device_attribute *attr, char *page)
          {
          	struct blk_integrity *bi = dev_to_bi(dev);
          
          	return sysfs_emit(page, "%d\n", !!(bi->flags & BLK_INTEGRITY_GENERATE));
          }
          
          static ssize_t device_is_integrity_capable_show(struct device *dev,
          						struct device_attribute *attr,
          						char *page)
          {
          	struct blk_integrity *bi = dev_to_bi(dev);
          
          	return sysfs_emit(page, "%u\n",
          			  !!(bi->flags & BLK_INTEGRITY_DEVICE_CAPABLE));
          }
          
          static DEVICE_ATTR_RO(format);

hsnyder · May 24, 2024, 5:08am

I believe my supposition that Type 1 DIF is correctly enabled (and it’s just DIX that isn’t) was correct.

Unfortunately, the reason I believe that is because the protection info seems to have caught a checksum mismatch. dmesg says the following. I don’t really have a point, I’m just noting it in case anyone has any further insight.

[  +0.002040] sd 0:0:1:0: [sdb] Spinning up disk...
[  +1.020276] .......................ready
[May24 00:39] sd 0:0:1:0: [sdb] 4848877568 4096-byte logical blocks: (19.9 TB/18.1 TiB)
[  +0.001091] sd 0:0:1:0: [sdb] Write Protect is off
[  +0.000008] sd 0:0:1:0: [sdb] Mode Sense: df 00 10 08
[  +0.001998] sd 0:0:1:0: [sdb] Write cache: enabled, read cache: enabled, supports DPO and FUA
[  +0.002706] sd 0:0:1:0: [sdb] Enabling DIF Type 1 protection
[  +0.000017] sd 0:0:1:0: [sdb] Preferred minimum I/O size 32768 bytes
[  +0.032329] sd 0:0:1:0: [sdb] Attached SCSI disk
[  +0.063584] mpt2sas_cm0: log_info(0x3112043b): originator(PL), code(0x12), sub_code(0x043b)
[  +0.000023] sd 0:0:1:0: [sdb] tag#2489 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=0s
[  +0.000005] sd 0:0:1:0: [sdb] tag#2489 Sense Key : Illegal Request [current] 
[  +0.000003] sd 0:0:1:0: [sdb] tag#2489 Add. Sense: Logical block reference tag check failed
[  +0.000004] sd 0:0:1:0: [sdb] tag#2489 CDB: Read(16) 88 20 00 00 00 01 21 03 ff df 00 00 00 01 00 00
[  +0.000003] protection error, dev sdb, sector 38791020280 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
[  +0.000293] mpt2sas_cm0: log_info(0x3112043b): originator(PL), code(0x12), sub_code(0x043b)
[  +0.000008] sd 0:0:1:0: [sdb] tag#2490 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=0s
[  +0.000003] sd 0:0:1:0: [sdb] tag#2490 Sense Key : Illegal Request [current] 
[  +0.000002] sd 0:0:1:0: [sdb] tag#2490 Add. Sense: Logical block reference tag check failed
[  +0.000003] sd 0:0:1:0: [sdb] tag#2490 CDB: Read(16) 88 20 00 00 00 01 21 03 ff df 00 00 00 01 00 00
[  +0.000001] protection error, dev sdb, sector 38791020280 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2
[  +0.000004] Buffer I/O error on dev sdb, logical block 4848877535, async page read

hsnyder · May 24, 2024, 5:48am

… upon further reading and thinking, if the “reference tag” check is failing, wouldn’t that mean that it’s not actually a checksum mismatch (that would be the “guard” tag, I think), but a misdirected read? Interesting. Perhaps the problem is not the disk itself…

twin_savage · May 24, 2024, 6:06am

That is how I was reading it.
I remember there being some kind of 4k block issue Linux had that would cause this, but for the life of me can’t remember where I read about it. It had something to do with a reftag remap being handled badly when going from the block input-output layer to the SCSI stack, the 4k bio>scsi remap had some emergent behavior that the 512b bio>scsi remap didn’t.

hsnyder · May 24, 2024, 6:10am

I barely follow to be honest, but I appreciate your reply. If you recall any additional details please let me know! I’m confused because this same HBA and same disk worked in a different host. Both hosts are running Debian 12, which is odd. I don’t know if the kernel versions were exactly the same but they must be close.

I turned up mpt3sas’s verbosity a bit and there’s a little more information, though it still doesn’t mean that much to me.

[  +0.000196] sd 0:0:0:0: [sdb] tag#257 CDB: Read(16) 88 20 00 00 00 00 02 4f e7 6c 00 00 00 01 00 00
[  +0.000003] mpt2sas_cm0:      sas_address(0x5000c500ec27b879), phy(3)
[  +0.000002] mpt2sas_cm0: enclosure logical id(0x500605b00535faa0), slot(0)
[  +0.000001] mpt2sas_cm0:      handle(0x0009), ioc_status(eedp ref tag error)(0x004e), smid(258)
[  +0.000002] mpt2sas_cm0:      request_len(4096), underflow(512), resid(0)
[  +0.000001] mpt2sas_cm0:      tag(1499), transfer_count(4096), sc->result(0x00050002)
[  +0.000001] mpt2sas_cm0:      scsi_status(good)(0x00), scsi_state( )(0x00)
[  +0.000003] mpt2sas_cm0: log_info(0x3112043b): originator(PL), code(0x12), sub_code(0x043b)
[  +0.000004] sd 0:0:0:0: [sdb] tag#257 FAILED Result: hostbyte=DID_ABORT driverbyte=DRIVER_OK cmd_age=0s
[  +0.000002] sd 0:0:0:0: [sdb] tag#257 Sense Key : Illegal Request [current] 
[  +0.000002] sd 0:0:0:0: [sdb] tag#257 Add. Sense: Logical block reference tag check failed
[  +0.000001] sd 0:0:0:0: [sdb] tag#257 CDB: Read(16) 88 20 00 00 00 00 02 4f e7 6c 00 00 00 01 00 00
[  +0.000002] protection error, dev sdb, sector 310328160 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 2

hsnyder · May 24, 2024, 6:49am

@twin_savage thanks for the tip! Based on my half-understanding of what you were saying, I had a hunch to go disable legacy option ROMs in the bios, thereby disabling the ROM for my SAS2008 chip. This seems to have fixed the issue! This also explains why this was working on my other host - that host has option ROMs disabled I believe. Thanks again

vlorentz · November 28, 2024, 12:37pm

@hsnyder: This is a very nice post and I am trying to do the same (i.e., enabling T10 PI/DIF/DIX). However, I did not get the point where you tested that both DIF and DIX are enabled. It seems that you managed to enable both on the Seagate Exos 20TB SAS drive: but how did you check that it is enabled? Were you able to display it as “enabled”? If yes, where and how? Furthermore, were you able to test it by injecting faults?
Best regards