CAM write errors zfs FreeBSD

Operating Systems & Open Source BSD
, ,
1

I’m writing a bunch of data to a zpool and noticed all these write errors on 2 disks (da0 and da2):

Dec 15 22:21:50 storeman mpr0: Controller reported scsi ioc terminated tgt 0 SMID 1411 loginfo 31120303
Dec 15 22:21:50 storeman (da0:mpr0:0:0:0): WRITE(10). CDB: 2a 00 0e 93 34 54 00 01 00 00 
Dec 15 22:21:50 storeman (da0:mpr0:0:0:0): CAM status: CCB request completed with an error
Dec 15 22:21:50 storeman (da0:mpr0:0:0:0): Retrying command, 3 more tries remain

Dec 15 22:21:53 storeman mpr0: Controller reported scsi ioc terminated tgt 0 SMID 1111 loginfo 31120303
Dec 15 22:21:53 storeman (da0:mpr0:0:0:0): WRITE(10). CDB: 2a 00 0e 93 5d 74 00 01 00 00 
Dec 15 22:21:53 storeman (da0:mpr0:0:0:0): CAM status: CCB request completed with an error
Dec 15 22:21:53 storeman (da0:mpr0:0:0:0): Retrying command, 3 more tries remain

I keep seeing these over and over and over again. Not sure what it means though. Is there an issue? TrueNAS isn’t saying anything’s wrong.

No ZFS errors for those drives (one is in each mirror):

image
image868×298 7.98 KB

dmesg looks like this:

(da2:mpr0:0:2:0): WRITE(10). CDB: 2a 00 0e 23 6f 5a 00 01 00 00
(da2:mpr0:0:2:0): CAM status: CCB request completed with an error
(da2:mpr0:0:2:0): Retrying command, 3 more tries remain
mpr0: Controller reported scsi ioc terminated tgt 0 SMID 1456 loginfo 31120303
(da0:mpr0:0:0:0): WRITE(10). CDB: 2a 00 0e dd 16 7e 00 01 00 00
(da0:mpr0:0:0:0): CAM status: CCB request completed with an error
(da0:mpr0:0:0:0): Retrying command, 3 more tries remain
mpr0: Controller reported scsi ioc terminated tgt 2 SMID 363 loginfo 31120303
(da2:mpr0:0:2:0): WRITE(10). CDB: 2a 00 0e 23 a6 3d 00 01 00 00
(da2:mpr0:0:2:0): CAM status: CCB request completed with an error
(da2:mpr0:0:2:0): Retrying command, 3 more tries remain
mpr0: Controller reported scsi ioc terminated tgt 0 SMID 796 loginfo 31120303
(da0:mpr0:0:0:0): WRITE(10). CDB: 2a 00 0f 1d b8 27 00 01 00 00
(da0:mpr0:0:0:0): CAM status: CCB request completed with an error
(da0:mpr0:0:0:0): Retrying command, 3 more tries remain
mpr0: Controller reported scsi ioc terminated tgt 2 SMID 117 loginfo 31120303
(da2:mpr0:0:2:0): WRITE(10). CDB: 2a 00 0e 23 b9 7d 00 00
2

I upgraded to TrueNAS SCALE and kept seeing these errors.

From my testing so far, it looks potentially like some problematic LSI 9305 cards. I went ahead and swapped them out and have yet to see these messages since.

They only seem to occur when reading or writing from the drives, so it’s hard to test since I need to be specifically interacting with those drives on those ports attached to those particular cards.