H12SSL-i PCIe Hot Plug support?

Hardware Hub Motherboards
1

I used to run a very old i5-3470 and PCIe Hotplug worked fine if you made use of

echo 1 > /sys/bus/pci/devices/0000:84:00.0/remove
echo 1 > /sys/bus/pci/rescan

before removing and after reattaching the the U.2 PCIe SSD. I use an Icydock MB699VP-B V3 connected to a SlimSAS Direct Attach riser.

However on my new Supermicro H12SSL-i, Epyc 7F72, 512GB 3200MT/s system, you can remove the SSD. However when hot plugging it back in, the system will hang and Linux will won’t throw any error in the log (aside from the previous hot removal’s AER error) or on the IPMI KVM. The IPMI log shows this error which I have searched online for but with not much success.

2024-06-20 00:01:58 Critical Interrupt [CI-0005] , PCI SERR @Bus80 (DevFn0C) - Assertion Sensor-specific

The Bus ID’s don’t line up as I have tried Hot-plugging a Samsung 970 Pro in a IcyDock M.2 to U.2 carrier and a Intel D7-P5520. No luck.

Can anyone help?

EDIT:

I am running Ubuntu Server 22.04.

I just tried disabling ACS and AER and I can hot plug with commands like before. Do I loose PCIe link error detection and correction if I leave AER off? What do setting under Linux do I need to change in order to use AER with Hot Plug?

2

Are you connecting the U.2 drive to the Slimsas connector on the board or to one of the slots? I think if you use the slots you might need one of the Supermicro AIC’s that has a retimer with an I2C cable going back to the mobo.

3

I have a riser card without a retimer from Linkreal with SlimSAS ports. The IcyDock “backplane” has no I2C management port. I also have a retimer card from the same manufacturer but It ended up unnecessary as AER showed no signal integrity errors.

I have since learned about acpiphp. Apparently in order to use AER, this needs to be configured or compiled into the kernel inorder for it to handle hotplug requests. I haven’t gotten further than that.

4

I just remember from my experience with first gen xeon scalable systems, that the supermicro AOC was required in order to get working hotplug.

You should also have the slimsas on the motherboard itself which is designed to be populated with NVMe devices. Perhaps you should try that and see if things work.

This is the Gen4 AOC for NVMe drives: ✅Supermicro AOC-SLG4-2E4T Dual Port Gen-4 Internal NVMe HBA, PCIe x8 Add-on Card | eBay

5

You could be right about the I2C cable being required for full HotPlug support.

modprobe acpiphp
modprobe pci_hotplug

This is what was missing and lets me enable AER without crashing. However unlike a video on LinusTechTips showing NVMe raid enclosures. I can’t simply yank a drive, insert another and have it detected. I still need the remove and rescan commands. Is that what the I2C cable is for? Hot Plug event detection.