[Solved] Troubleshooting PCIE errors on HP DL380 Gen10

High-Performance Computing Workstations & Servers
1

Hi,

I’m trying to upgrade an HP DL380 Gen10 for use as a visualisation server. RAM and CPU upgrades went fine, but stuck trying to add fast storage. I’m attempting to add NVMe storage using a Silverstone ECM40 riser and PCIE bifurcation.

The issue is that as soon as an nvme drive is installed in the riser, the server refuses to boot. iLO errors indicate a crc fail, and it won’t even go into bios setup without disabling the port.

I’ve tried changing between Auto/Dual Bifurcate on the slot, slowed down the PCIE bus to pcie2.0, swapped the NVME to other slots on the riser, nothing helps.

The riser works fine with one drive in another machine (no PCIE bifurcation, so can’t try more). Im using slot 5, attached to cpu2, because the nvme riser is too high to put into the primary PCIE riser position on the server, so I can’t try in a different slot with the same riser.

Any ideas to get it working, or at least work out where the problem lies?

2

Is the ECM40 you use a older or newer version?
Older hardware revision had a exposed i2c bus (Most probable) or smbus from the cards power ic feeding back in the pcie slot and just jamming the ILO control channels as it’s also i2c

3

Never mind, I’m not able to find the post I was thinking about.
So it might not have been the ECM40

Still you might want check of no i2c or smbus is interfering.
There is a simple mod you can try, you need to tape off pin B5 and B6 on the ECM40 Guide

4

That’s a really good idea, unfortunately I taped off the relevant pins and it made no difference.

I found a 1-drive riser in a cupboard (one of the low-cost ones from ebay with almost no electronics on it), and tested in multiple slots, any time the nvme drive is plugged in the machine faults before completing post with a CRC error on the slot that the nvme drive is plugged into.

The nvme drives should be backwards compatible because they worked in another machine with a pcie3.0 slot. Could it be an issue from the HP side with the drive bus negotiations? The SSD is a Lexar NM990.

5

The specific error from iLO is:

Uncorrectable PCI Express Error Detected. Slot 6 (Segment 0x0, Bus 0xAE, Device 0x2, Function 0x0). Uncorrectable Error Status: 0x4000

6

I think I found a solution. Disabling power management and mctp broadcast on that pcie port allows the server to boot successfully. I’m guessing the drive didn’t like something about the boot process, and simplifying the pcie bus options helped it power up ok.

Thanks very much for your help.

7

My pleasure, glad you got it running.
HPE can be iffy if not all hardware in the chain is labeled HP.
But it might also be the lack off a specific power state or mctp in any of the SSD’s.

BTW if you notice your fans are running louder than before, also a HP thing.
If any hardware is not supported in ILO (Just about all non HP branded parts) the default action is to ramp up the fans.
I seen it happen on a bunch of DL20 gen10s at least those almost took off when a generic drive was inserted, I hope your DL380 is more flexible.