LSI 9600 24i no TRIM support for sata SSD?

Hardware Hub Storage
1

Hi,

i’m using a LSI 9600 24i HBA on a Threadripper 5955WX + Asus Sage WRX80 motherboard to connect 24x1.92TB SSDs.
The system is running Proxmox and the card is passed to a Truenas Scale VM.
The drives are a mix of enterprise Samsung, Seagate and Micron SSD, hdparm show that they support “Deterministic read ZEROs after TRIM” and trim worked with a 9300 card.
I updated to the latest firmware, i have trouble to understand all of the personality configuration but it is configured as eHBA and JBOD, i tried the latest nightly of Truenas Scale to see if a newer driver might help without succès.

This is the result :

sudo sudo zpool status -t nandy pool: nandy
state: ONLINE
scan: resilvered 588G in 10:23:29 with 0 errors on Thu Apr 4 23:46:51 2024
config:

    NAME                                      STATE     READ WRITE CKSUM
    nandy                                     ONLINE       0     0     0
      raidz1-0                                ONLINE       0     0     0
        d04bd7ad-7a4b-4217-bc4e-72a4b646b3f5  ONLINE       0     0     0  (trim unsupported)
        51c7579f-0f0c-4381-9575-6b9b9b274fc2  ONLINE       0     0     0  (trim unsupported)
        73edb1f4-33cb-41c2-b621-4ebc043cb972  ONLINE       0     0     0  (trim unsupported)
        d5336649-911a-4979-9f85-6486793e9f06  ONLINE       0     0     0  (trim unsupported)
        2bf39951-917c-4a2b-aa0d-bf3b742b90b5  ONLINE       0     0     0  (trim unsupported)
        7cafa449-25ef-4557-863b-97b827a87fb5  ONLINE       0     0     0  (trim unsupported)
        4cc38f60-a443-440c-8189-43f8f00526bf  ONLINE       0     0     0  (trim unsupported)
        398d42c7-7be5-412e-824a-401ec708904d  ONLINE       0     0     0  (trim unsupported)
        0cb09409-904a-49c8-a733-b9bc886f1339  ONLINE       0     0     0  (trim unsupported)
        d2ea36ec-f63a-4394-b847-8abc2e5664c7  ONLINE       0     0     0  (trim unsupported)
        e3744651-8741-4085-aabe-d90f19e98e24  ONLINE       0     0     0  (trim unsupported)
        f2e8e434-ff1a-421a-a896-459db0ee4c7e  ONLINE       0     0     0  (trim unsupported)
      raidz1-1                                ONLINE       0     0     0
        c8ef7310-8b04-466d-9052-1efdc89ed6eb  ONLINE       0     0     0  (trim unsupported)
        33206209-c237-497c-b2a1-456df91e7ce0  ONLINE       0     0     0  (trim unsupported)
        f2d32b55-7220-485f-acac-d83ad9a798c0  ONLINE       0     0     0  (trim unsupported)
        04fae63e-51b6-437b-b4da-825957248502  ONLINE       0     0     0  (trim unsupported)
        eedc0d38-962e-43e6-a8b4-0ab8e4877df2  ONLINE       0     0     0  (trim unsupported)
        1124b472-3871-4d3b-883b-68028cc70615  ONLINE       0     0     0  (trim unsupported)
        a283d8eb-7dbb-4b5b-ae6c-d6a4dc7f7453  ONLINE       0     0     0  (trim unsupported)
        9e4b651c-b328-4ff3-8713-b8a7cd0ed6cc  ONLINE       0     0     0  (trim unsupported)
        f87619ee-15a4-486a-9694-91cd52bbea02  ONLINE       0     0     0  (trim unsupported)
        b09429d3-d276-4591-9935-d33ea64677dd  ONLINE       0     0     0  (trim unsupported)
        66cd7a7c-01f1-40cd-89be-8293079d7663  ONLINE       0     0     0  (trim unsupported)
        b2ebaab4-e80c-4eaa-8726-24d0eaf6ef94  ONLINE       0     0     0  (trim unsupported)

At first this pool was as fast as expected, now it’s at most a few hundreds mb/s write speed, reads are normal.

Does anyone have experience with this card ? Is there something to configure ?

I have read a lot about this card here and elsewhere before buiying it, i’m aware that it is kind of shit with nvme, but i have never seen anybody mention problems with sata, trim or basic functionality like that, maybe i am missing something ?

2

Found this in a google search:

The drives must support both “Data Set Management TRIM supported (limit 8 blocks)” and “Deterministic read ZEROs after TRIM” in their ATA options.
The Samsung 850 PROs don’t have “Deterministic read ZEROs after TRIM” support, and thus TRIM cannot be run on these drives when attached to a LSI SAS HBAs with IT firmware

You can also use sg_unmap to send a SCSI UNMAP command. Sg_unmap is part of the sg3_utils, which can be downloaded from here: The sg3_utils package

Usage: sg_unmap [–grpnum=GN] [–help] [–in=FILE] [–lba=LBA,LBA…]
[–num=NUM,NUM…] [–timeout=TO] [–verbose] [–version] DEVICE

Send a SCSI UNMAP command to DEVICE to unmap one or more logical blocks. The unmap capability is closely related to the ATA DATA SET MANAGEMENT command with the “Trim” bit set. Click here for a more detailed description: http://manpages.ubuntu.com/manpages/lucid/man8/sg_unmap.8.html

So maybe your SSDs dont support the second part of TRIM to work when connected to an HBA? Maybe SG_UNMAP can work for you?

You also need IT mode firmware, but it sounds like you already have tat if the drives are JBOD.

1 Like
3

Thanks for the reply, the drives have all the features necessary, i know of some consumer drives who don’t work on HBAs but i use SSD like Micron 5400 Max and Samsung PM893, i don’t think the problem is there.

UNMAP seems it might do the job but i’m not sure i understand correctly, if i send the command to a whole disk with some data on it, does it work exacly like TRIM ? I don’t have to specify witch blocs are to be “unmaped” and my data is safe ?

There is also an UNMAP command in storecli2 but it seems to apply only to virtual volumes, i didn’t realised it was also aviable on the OS CLI, thanks for that !

https://techdocs.broadcom.com/us/en/storage-and-ethernet-connectivity/enterprise-storage-solutions/megaraid8-tri-mode-software/1-0/StorCLI_MR8_Intro/StorCLI-to-StorCLI2-Command-Map.html

On the 9600 series the firmware is different, there is no more IT mode flashing, just different “personality” to choose from in the configuration.