Lsi 9600-24i required cable do they even work? 8654 to 8087

sy5tem · March 9, 2024, 12:08am

Hi,

i got a 9600-24i, learned that it would work only in truenas dragonfish, so i loaded it. nope i can’t see drives.

i can see the card in lspci…

im wondering if my cable (all 3 ) are bad? i never tried connecting a 9600 to a 8087 backplane before

loaded windows and see the card but no the drive?

is there there some magic sauce to tell the card that is in regular sas drive mode instead of nvme?

ordered a 8654 to 8x sata to test later…

just asking for experienced person who had dealing with this kind of thing before!

thank you

card show working i guess?
C:\Users\test\Downloads\StorCLI_Avenger_8.8-008.0008.0000.0010\StorCLI_Avenger_8.8-008.0008.0000.0010\Avenger_StorCLI\Windows>storcli2.exe /c0 show
CLI Version = 008.0008.0000.0010 Jan 08, 2024
Operating system = Windows 10
Controller = 0
Status = Success
Description = None

Product Name = eHBA 9600-24i Tri-Mode Storage Adapter
Board Name = eHBA 9600-24i
Board Assembly = 03-50111-01001
Board Tracer Number = SPC2607401
Board Revision = 01001
Chip Name = SAS4024
Chip Revision = B0
Package Version = 8.1.1.0-00000-00001
Firmware Version = 8.1.1.0-00000-00001
Firmware Security Version Number = 00.00.00.00
NVDATA Version = 01.7E.00.39
Driver Name = mpi3drv.sys
Driver Version = 8.08.07.00
SAS Address = 0x500062b20e1f4e80
Serial Number = SPC2607401
Controller Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/08 16:12:27
System Time(LocalTime yyyy/mm/dd hh:mm:sec) = 2024/03/08 16:12:27
Board Mfg Date(yyyy/mm/dd) = 2022/07/11
Controller Personality = eHBA Personality
Max PCIe Link Rate = 0x08 (16GT/s)
Max PCIe Port Width = 8
PCI Address = 00:81:00:0
PCIe Link Width = X8 Lane(s)
SAS/SATA = SAS/SATA-6G, SAS-12G, SAS-22.5G
PCIe = PCIE-2.5GT, PCIE-5GT, PCIE-8GT, PCIE-16GT
PCI Vendor ID = 0x1000
PCI Device ID = 0x00A5
PCI Subsystem Vendor ID = 0x1000
PCI Subsystem ID = 0x4660
Security Protocol = SPDM-1.1.0,1.0.0
PCI Slot Number = 801

cowphrase · March 9, 2024, 1:23am

Get into the card’s option rom at boot, and see if the drives show up there. I’m not familiar with the card, but it may be in RAID mode instead of HBA mode?

sy5tem · March 9, 2024, 7:23pm

Hi,

thank for reply,

the card as only 1 personality i think “checked with the storcli2 command” and its hba.

i am also very unfamiliar with this card, i played a bunch with raid card since my 3ware/areca and even some scsi cards in my days. im also used to have some kind of usable gui at boot or even in uefi bios.

this card is invisible at boot and not shown in bios. updated to latest bios all i could do , waiting on 8654 to 8x sata cable to test…

might just be that i bought the wrong cable? i think since this card is “tri-mode” you probably need cable especially made for 8654 to sas back plane meaby?
dunno

thanks

twin_savage · March 9, 2024, 7:28pm

Is the backplane UBM, SAS expander or is it straight shot through? Also what kind of drives (SATA/SAS/NVME U.2/NVME U.3) are you trying to run?
Different types of cables are necessary depending on your combo of answers.

sy5tem · March 9, 2024, 10:35pm

Hi,

its a SAS backplane with SAS hdd.

used to work with a 9305-24i (but doesn’t fit in case)

cable im using now

and here is one of the backplane qumulo QC208:

hope you know what cable are compatible if any!!!

thank you

twin_savage · March 10, 2024, 5:27am

Hmmm SAS drives connected with just a “straight through” backplane, the cable you bought should have been the correct one.
It’s possible there is some configuration required on the 9600-24i before it connects properly.

cowphrase · March 10, 2024, 5:35am

It should show a screen just before the OS when booting up showing “Loading firmware” and “Hit Ctrl + _ to configure”. If it doesn’t load try disabling secure boot, then booting in CSM mode.

Also this thread may provide some more info.

z8z · March 13, 2024, 7:55pm

I’m currently in the same situation. My 9600-24i is on the latest firmware. I can get the card to show up in TrueNAS Scale, and on the latest beta it even loads the mpi3mr driver. None of my drives are showing up though.

With that said, if I swap my TrueNAS SSD for an Ubuntu install I can see the disks just fine. fdisk -l spits out all of my drives, so the cable is definitely not at fault here and neither is the card.

Edit: let me provide some more information to clarify.

I have six shucked 18TB Western Digital white label drives plugged into a SAS backplane, connected with a SFF-8654 to 2x SFF-8087 cable I bought on Amazon. The HBA I was using up until this point was a 9305-24i which had two SFF-8643 to SFF-8087 cables plugged into it.

z8z · March 14, 2024, 3:56pm

Update:

dmesg spits out the following on the TrueNAS Beta:

root@truenas[/home/admin]# dmesg | grep mpi3mr
[    4.937280] Loading mpi3mr version 8.5.0.0.0
[    4.937298] mpi3mr 0000:06:00.0: osintfc_mrioc_security_status: PCI_EXT_CAP_ID_DSN is not supported
[    4.937302] mpi3mr 0000:06:00.0: Driver probe function unexpectedly returned 1

Here’s the line in the mpi3mr driver fo it:

github.com

torvalds/linux/blob/480e035fc4c714fb5536e64ab9db04fedc89e910/drivers/scsi/mpi3mr/mpi3mr_os.c#L4989


      
          */
          static int
          osintfc_mrioc_security_status(struct pci_dev *pdev)
          {
          	u32 cap_data;
          	int base;
          	u32 ctlr_status;
          	u32 debug_status;
          	int retval = 0;
          
          	base = pci_find_ext_capability(pdev, PCI_EXT_CAP_ID_DSN);
          	if (!base) {
          		dev_err(&pdev->dev,
          		    "%s: PCI_EXT_CAP_ID_DSN is not supported\n", __func__);
          		return -1;
          	}
          
          	pci_read_config_dword(pdev, base + 4, &cap_data);
          
          	debug_status = cap_data & MPI3MR_CTLR_SECURE_DBG_STATUS_MASK;
          	ctlr_status = cap_data & MPI3MR_CTLR_SECURITY_STATUS_MASK;

Ubuntu on the other hand gives me the following upon boot. This time around the drives are working correctly.

root@z8-epyc:/home/z8# dmesg | grep mpi3mr
[    3.980011] Loading mpi3mr version 8.5.0.0.0
[    3.983065] mpi3mr0: mpi3mr_probe :host protection capabilities enabled  DIF1 DIF2 DIF3
[    3.983441] mpi3mr0: iomem(0x0000000051200000), mapped(0x00000000736d2db7), size(16384)
[    3.983445] mpi3mr0: Number of MSI-X vectors found in capabilities: (128)
[    3.983451] mpi3mr0: ioc_status(0x00000011), ioc_config(0x00470001), ioc_info(0x00000000ff000000) at the bringup
[    3.983453] mpi3mr0: ready timeout: 510 seconds
[    3.983456] mpi3mr0: controller is in ready state during detection
[    3.983458] mpi3mr0: issuing message unit reset (MUR) to bring to reset state
[    3.983458] mpi3mr0: Issuing Message unit Reset(MUR)
[    4.304785] mpi3mr0: Base IOC Sts/Config after successful MUR is (0x10)/(0x470000)
[    4.304803] mpi3mr0: bringing controller to ready state
[    4.412763] mpi3mr0: successfully transitioned to ready state
[    4.413221] mpi3mr0: ioc_num(0), maxopQ(127), maxopRepQ(127), maxdh(1023),
[    4.413238] mpi3mr0: maxreqs(8192), mindh(1) maxvectors(128) maxperids(1024)
[    4.413239] mpi3mr0: SGEModMask 0x80 SGEModVal 0x80 SGEModShift 0x18
[    4.413240] mpi3mr0: DMA mask 63 InitialPE status 0x20 max_data_len (1048576)
[    4.413241] mpi3mr0: max_dev_per_throttle_group(0), max_throttle_groups(0)
[    4.413242] mpi3mr0: io_throttle_data_len(0KiB), io_throttle_high(0MiB), io_throttle_low(0MiB)
[    4.413243] mpi3mr0: Changing DMA mask from 0xffffffffffffffff to 0x7fffffffffffffff
[    4.413245] mpi3mr0: Running in Enhanced HBA Personality
[    4.413246] mpi3mr0: FW version(8.8.1.0.0.1)
[    4.413247] mpi3mr0: Protocol=(Initiator,NVMe attachment), Capabilities=(RAID,MultiPath)
[    4.413563] mpi3mr0: number of sgl entries=256 chain buffer size=4KB
[    4.415287] mpi3mr0: reply buf pool(0x00000000d210bbe1): depth(8256), frame_size(128), pool_size(1032 kB), reply_dma(0x10ce00000)
[    4.415291] mpi3mr0: reply_free_q pool(0x00000000435a3c4a): depth(8257), frame_size(8), pool_size(64 kB), reply_dma(0x10ccc0000)
[    4.415293] mpi3mr0: sense_buf pool(0x000000008f8ea92e): depth(2730), frame_size(256), pool_size(682 kB), sense_dma(0x10cd00000)
[    4.415294] mpi3mr0: sense_buf_q pool(0x00000000eac2ea28): depth(2731), frame_size(8), pool_size(21 kB), sense_dma(0x10bfd0000)
[    4.417625] mpi3mr0: firmware package version(8.8.1.0.00000-00001)
[    4.417811] mpi3mr0: MSI-X vectors supported: 128, no of cores: 8,
[    4.417813] mpi3mr0: MSI-x vectors requested: 9 poll_queues 0
[    4.418380] mpi3mr0: trying to create 8 operational queue pairs
[    4.418385] mpi3mr0: allocating operational queues through segmented queues
[    4.430368] mpi3mr0: successfully created 8 operational queue pairs(default/polled) queue = (8/0)
[    4.430852] mpi3mr0: controller initialization completed successfully
[    4.431701] mpi3mr0: mpi3mr_scan_start :Issuing Port Enable
[    4.434273] mpi3mr0: Enclosure Added
[    4.434672] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.434782] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.434801] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.434821] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.434839] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.434860] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.434878] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.434899] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.434917] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.434937] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.434955] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.434976] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.434994] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.435013] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.435032] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.435052] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.435095] mpi3mr0: PCIE Enumeration: (start)
[    4.435124] mpi3mr0: PCIE Enumeration: (stop)
[    4.435143] mpi3mr0: PCIE Enumeration: (start)
[    4.435172] mpi3mr0: PCIE Enumeration: (stop)
[    4.435190] mpi3mr0: PCIE Enumeration: (start)
[    4.435220] mpi3mr0: PCIE Enumeration: (stop)
[    4.435238] mpi3mr0: PCIE Enumeration: (start)
[    4.435267] mpi3mr0: PCIE Enumeration: (stop)
[    4.435285] mpi3mr0: PCIE Enumeration: (start)
[    4.435315] mpi3mr0: PCIE Enumeration: (stop)
[    4.435333] mpi3mr0: PCIE Enumeration: (start)
[    4.435363] mpi3mr0: PCIE Enumeration: (stop)
[    4.439491] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.439512] mpi3mr0: Device Added: dev=0x0007 Form=0x0
[    4.439532] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.439540] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.439559] mpi3mr0: Device Added: dev=0x0009 Form=0x0
[    4.439579] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.439586] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.439606] mpi3mr0: Device Added: dev=0x0005 Form=0x0
[    4.439626] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.439633] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.439654] mpi3mr0: Device Added: dev=0x0006 Form=0x0
[    4.439675] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.439681] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.439701] mpi3mr0: Device Added: dev=0x0003 Form=0x0
[    4.439721] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.439728] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.439748] mpi3mr0: Device Added: dev=0x0008 Form=0x0
[    4.439766] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.439784] mpi3mr0: Device Added: dev=0x0004 Form=0x0
[    4.448757] mpi3mr0: port enable is successfully completed

I’d do more troubleshooting if I had the time but that will have to wait until this weekend.

Gggopkmsw · March 15, 2024, 2:27pm

I recently got one of these 9600-24i and am still working through which cables and which backplane. Or if some sort of midplane exists to further bifurcate some of the lanes.

I’d seen some 8654 x8 cables that take 7 sata drives. Or something similar, might be an x4 to 7sata drives. I have some cables at home that I think have 8654 to host and SAS NVME connections to drives.

I’m basically trying to run 4 NVME SSD drives, if I can connect the sata SSD drives that just a bonus. I’m also using some old icydock drive bays 6 + 1 m.2 adapter and 8 drive tough armor as a backplane. They’re all sata connections. After buying the card I realized if I want another icydock drive cage for this it’s near or around $400 (or is a concept product in development), so isn’t in the budget or has yet to be made. So I’m hoping this can all be configured with just connecting the drives to the card.

TLDR; I appreciate your testing different cables and all. Look forward to seeing what you get working!

z8z · March 15, 2024, 3:53pm

Excellent news: TrueNAS on the latest nightly build I could find (in this case ElectricEel-24.10.0-MASTER-20240302-032825) can see my drives!

root@truenas[/home/admin]# uname -a
Linux truenas 6.6.16-production+truenas #1 SMP PREEMPT_DYNAMIC Sat Mar  2 09:11:19 UTC 2024 x86_64 GNU/Linux
root@truenas[/home/admin]# dmesg | grep mpi3mr
[    4.299173] Loading mpi3mr version 8.5.0.0.0
[    4.299267] mpi3mr0: mpi3mr_probe :host protection capabilities enabled  DIF1 DIF2 DIF3
[    4.299621] mpi3mr0: iomem(0x0000000051200000), mapped(0x00000000ac7667f5), size(16384)
[    4.299623] mpi3mr0: Number of MSI-X vectors found in capabilities: (128)
[    4.299628] mpi3mr0: ioc_status(0x00000011), ioc_config(0x00470001), ioc_info(0x00000000ff000000) at the bringup
[    4.299629] mpi3mr0: ready timeout: 510 seconds
[    4.299632] mpi3mr0: controller is in ready state during detection
[    4.299633] mpi3mr0: issuing message unit reset (MUR) to bring to reset state
[    4.299633] mpi3mr0: Issuing Message unit Reset(MUR)
[    4.732690] mpi3mr0: Base IOC Sts/Config after successful MUR is (0x10)/(0x470000)
[    4.732968] mpi3mr0: bringing controller to ready state
[    4.840696] mpi3mr0: successfully transitioned to ready state
[    4.840946] mpi3mr0: ioc_num(0), maxopQ(127), maxopRepQ(127), maxdh(1023),
[    4.840948] mpi3mr0: maxreqs(8192), mindh(1) maxvectors(128) maxperids(1024)
[    4.840949] mpi3mr0: SGEModMask 0x80 SGEModVal 0x80 SGEModShift 0x18
[    4.840950] mpi3mr0: DMA mask 63 InitialPE status 0x20 max_data_len (1048576)
[    4.840952] mpi3mr0: max_dev_per_throttle_group(0), max_throttle_groups(0)
[    4.840952] mpi3mr0: io_throttle_data_len(0KiB), io_throttle_high(0MiB), io_throttle_low(0MiB)
[    4.840954] mpi3mr0: Changing DMA mask from 0xffffffffffffffff to 0x7fffffffffffffff
[    4.840955] mpi3mr0: Running in Enhanced HBA Personality
[    4.840956] mpi3mr0: FW version(8.8.1.0.0.1)
[    4.840958] mpi3mr0: Protocol=(Initiator,NVMe attachment), Capabilities=(RAID,MultiPath)
[    4.841391] mpi3mr0: number of sgl entries=256 chain buffer size=4KB
[    4.843053] mpi3mr0: reply buf pool(0x00000000ec8c5394): depth(8256), frame_size(128), pool_size(1032 kB), reply_dma(0x100a00000)
[    4.843055] mpi3mr0: reply_free_q pool(0x0000000053f96675): depth(8257), frame_size(8), pool_size(64 kB), reply_dma(0x100900000)
[    4.843056] mpi3mr0: sense_buf pool(0x0000000069d34c02): depth(2730), frame_size(256), pool_size(682 kB), sense_dma(0x100c00000)
[    4.843057] mpi3mr0: sense_buf_q pool(0x000000004b4636cd): depth(2731), frame_size(8), pool_size(21 kB), sense_dma(0x100810000)
[    4.844896] mpi3mr0: firmware package version(8.8.1.0.00000-00001)
[    4.903168] mpi3mr0: MSI-X vectors supported: 128, no of cores: 8,
[    4.903170] mpi3mr0: MSI-x vectors requested: 9 poll_queues 0
[    4.904132] mpi3mr0: trying to create 8 operational queue pairs
[    4.904134] mpi3mr0: allocating operational queues through segmented queues
[    4.915272] mpi3mr0: successfully created 8 operational queue pairs(default/polled) queue = (8/0)
[    4.916057] mpi3mr0: controller initialization completed successfully
[    4.916805] mpi3mr0: mpi3mr_scan_start :Issuing Port Enable
[    4.918883] mpi3mr0: Enclosure Added
[    4.919329] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919444] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919463] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919493] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919521] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919550] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919579] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919608] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919636] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919669] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919698] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919732] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919761] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919794] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919822] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.919842] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.919892] mpi3mr0: PCIE Enumeration: (start)
[    4.919926] mpi3mr0: PCIE Enumeration: (stop)
[    4.919955] mpi3mr0: PCIE Enumeration: (start)
[    4.919989] mpi3mr0: PCIE Enumeration: (stop)
[    4.920018] mpi3mr0: PCIE Enumeration: (start)
[    4.920052] mpi3mr0: PCIE Enumeration: (stop)
[    4.920081] mpi3mr0: PCIE Enumeration: (start)
[    4.920115] mpi3mr0: PCIE Enumeration: (stop)
[    4.920144] mpi3mr0: PCIE Enumeration: (start)
[    4.920180] mpi3mr0: PCIE Enumeration: (stop)
[    4.920208] mpi3mr0: PCIE Enumeration: (start)
[    4.920243] mpi3mr0: PCIE Enumeration: (stop)
[    4.924371] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.924400] mpi3mr0: Device Added: dev=0x0005 Form=0x0
[    4.924430] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.924437] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.924466] mpi3mr0: Device Added: dev=0x0009 Form=0x0
[    4.924496] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.924503] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.924532] mpi3mr0: Device Added: dev=0x0007 Form=0x0
[    4.924561] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.924568] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.924597] mpi3mr0: Device Added: dev=0x0008 Form=0x0
[    4.924626] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.924634] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.924662] mpi3mr0: Device Added: dev=0x0006 Form=0x0
[    4.924691] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.924698] mpi3mr0: SAS Discovery: (start) status (0x00000000)
[    4.924728] mpi3mr0: Device Added: dev=0x0003 Form=0x0
[    4.924757] mpi3mr0: SAS Discovery: (stop) status (0x00000000)
[    4.924786] mpi3mr0: Device Added: dev=0x0004 Form=0x0
[    4.936698] mpi3mr0: port enable is successfully completed

I have two U.2 Optane 905P drives that I could theoretically connect to this card for testing, but I don’t have any U.2 to SFF-8654 cables.

sy5tem · March 19, 2024, 3:50pm

in my case, i just tested my cable at my office, none of the backplane drives ares detected, tried hp / supermicro / lenovo backplane, card reports no drives,

i tried a 8654 to 8x sata, and card detect drives. so in my case i have bad cable i guess, i try andg et same cable a z8z here .

sy5tem · March 22, 2024, 7:11pm

well with cable z8z has its working … but one is too shorti did not check b4 buying lol

so on a quest to find a 1m cable now i just don’t know where to find one!

edit:
found some at arecashop.com, there goes 200$ CND with shipping for 2 cable argh!