Hello all.
We bought disks to extend our ZFS pool in our off-site backup server. There is a pair of NVMe Samsung PM9A3 PN MZQL23T8HCLS-00A07 that do not work when connected to HPe 774741-B21, which uses a PCIe Gen3 switch with PEX 8749. It is in a DL380 Gen9 server.
The PEX8749 works fine with a pair of Intel P4500, never had any issues since the server was commissioned in 2019. The Intel disks have PCIe Gen3 interface, while the new Samsung PM9A3 have PCIe Gen4 interface.
I suspect some sort of compatibility issue between the Gen4 disks and Gen3 switch.
The Samsung PM9A3 disks are not even visible in BIOS or lspci while the Intel P4500 are visible.
When connecting the Samsung PM9A3 to the running host, the following appears in dmesg:
[193268.871128] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Attention button pressed
[193268.871139] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Powering off due to button press
[193268.871148] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Card not present
[193268.871152] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Already disabled
[193268.871157] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Card present
[193271.332276] pcieport 0000:06:0a.0: pciehp: Slot(0-1): No link
[193295.732655] pcieport 0000:06:09.0: pciehp: Slot(0): Attention button pressed
[193295.732666] pcieport 0000:06:09.0: pciehp: Slot(0): Powering off due to button press
[193295.732678] pcieport 0000:06:09.0: pciehp: Slot(0): Card not present
[193295.732683] pcieport 0000:06:09.0: pciehp: Slot(0): Already disabled
[193295.732688] pcieport 0000:06:09.0: pciehp: Slot(0): Card present
[193298.224616] pcieport 0000:06:09.0: pciehp: Slot(0): No link
[193303.795926] pcieport 0000:06:09.0: pciehp: Slot(0): Attention button pressed
[193303.795948] pcieport 0000:06:09.0: pciehp: Slot(0) Powering on due to button press
[193303.796708] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Attention button pressed
[193303.796715] pcieport 0000:06:0a.0: pciehp: Slot(0-1) Powering on due to button press
[193308.860806] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Card present
[193308.860814] pcieport 0000:06:09.0: pciehp: Slot(0): Card present
[193311.336772] pcieport 0000:06:0a.0: pciehp: Slot(0-1): No link
[193311.356776] pcieport 0000:06:09.0: pciehp: Slot(0): No link
I was not able to validate that the new disks work in another (preferably PCIe Gen4 system) since I have no access to such a system or needed cables/adapter cards. From dmesg it seems the disk ‘talks’ to the PCIe Gen3 switch when plugged in but something goes wrong.
It is Ubuntu 22.04. Kernel 5.15.0-112-generic.
The PEX8749 card sits in a x16 PCIe Gen3 slot on a riser. We have tried two different slots, same result. There are 6 U.2 bays with PCIe Ge3 x4 coming from the PEX8749.
This appears in dmesg during boot when the Samsung disks are not plugged in, only the functional Intel disks.
[ 7.555606] pci 0000:06:09.0: [10b5:8749] type 01 class 0x060400
[ 7.555721] pci 0000:06:09.0: PME# supported from D0 D3hot D3cold
[ 7.555804] pci 0000:06:0a.0: [10b5:8749] type 01 class 0x060400
[ 7.555917] pci 0000:06:0a.0: PME# supported from D0 D3hot D3cold
[ 7.556869] pci 0000:06:09.0: PCI bridge to [bus 08]
[ 7.556909] pci 0000:06:0a.0: PCI bridge to [bus 09]
[ 7.626782] pci 0000:06:09.0: bridge window [io 0x1000-0x0fff] to [bus 08] add_size 1000
[ 7.626789] pci 0000:06:09.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 08] add_size 200000 add_align 100000
[ 7.626796] pci 0000:06:09.0: bridge window [mem 0x00100000-0x000fffff] to [bus 08] add_size 200000 add_align 100000
[ 7.626802] pci 0000:06:0a.0: bridge window [io 0x1000-0x0fff] to [bus 09] add_size 1000
[ 7.626806] pci 0000:06:0a.0: bridge window [mem 0x00100000-0x000fffff 64bit pref] to [bus 09] add_size 200000 add_align 100000
[ 7.626812] pci 0000:06:0a.0: bridge window [mem 0x00100000-0x000fffff] to [bus 09] add_size 200000 add_align 100000
[ 7.627010] pci 0000:06:09.0: BAR 14: no space for [mem size 0x00200000]
[ 7.627014] pci 0000:06:09.0: BAR 14: failed to assign [mem size 0x00200000]
[ 7.627018] pci 0000:06:09.0: BAR 15: assigned [mem 0x38000000000-0x380001fffff 64bit pref]
[ 7.627023] pci 0000:06:0a.0: BAR 14: no space for [mem size 0x00200000]
[ 7.627026] pci 0000:06:0a.0: BAR 14: failed to assign [mem size 0x00200000]
[ 7.627030] pci 0000:06:0a.0: BAR 15: assigned [mem 0x38000200000-0x380003fffff 64bit pref]
[ 7.627065] pci 0000:06:09.0: BAR 13: assigned [io 0x4000-0x4fff]
[ 7.627068] pci 0000:06:0a.0: BAR 13: assigned [io 0x5000-0x5fff]
[ 7.627102] pci 0000:06:0a.0: BAR 14: no space for [mem size 0x00200000]
[ 7.627105] pci 0000:06:0a.0: BAR 14: failed to assign [mem size 0x00200000]
[ 7.627108] pci 0000:06:09.0: BAR 14: no space for [mem size 0x00200000]
[ 7.627111] pci 0000:06:09.0: BAR 14: failed to assign [mem size 0x00200000]
[ 7.627145] pci 0000:06:09.0: PCI bridge to [bus 08]
[ 7.627148] pci 0000:06:09.0: bridge window [io 0x4000-0x4fff]
[ 7.627166] pci 0000:06:09.0: bridge window [mem 0x38000000000-0x380001fffff 64bit pref]
[ 7.627182] pci 0000:06:0a.0: PCI bridge to [bus 09]
[ 7.627186] pci 0000:06:0a.0: bridge window [io 0x5000-0x5fff]
[ 7.627203] pci 0000:06:0a.0: bridge window [mem 0x38000200000-0x380003fffff 64bit pref]
[ 7.661816] pcieport 0000:06:09.0: pciehp: Slot #0 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd+ HotPlug+ Surprise- Interlock- NoCompl- IbPresDis- LLActRep+
[ 7.661965] pcieport 0000:06:09.0: pciehp: Slot(0): Card present
[ 7.661979] pcieport 0000:06:09.0: pciehp: Slot(0): Already enabled
[ 7.662366] pcieport 0000:06:0a.0: pciehp: Slot #0 AttnBtn+ PwrCtrl+ MRL- AttnInd- PwrInd+ HotPlug+ Surprise- Interlock- NoCompl- IbPresDis- LLActRep+
[ 7.662494] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Card present
[ 7.662508] pcieport 0000:06:0a.0: pciehp: Slot(0-1): Already enabled
I will be happy to receive any advice how to fix or troubleshoot this issue.