Connectx-4 Lx 2x 25GbE with ASPM support + idle power consumption measurements

High-Performance Computing Workstations & Servers
,
1

I’ve recently got myself 2 Connectx-4 Lx cards. I wanted to replace very sketchy connectx-3 cards. (OCP2.0 cards on pcie adapters which negotiate only pcie3x2 D:)
I’ve found decent cards on aliexpress for 62 EUR without taxes. It’s a steal at that price.

Out of the box card didn’t support ASPM. I’ve seen some people on servethehome talking about ASPM support being firmware dependant. Thankfully my cards were not OEM locked and I’ve updated the firmware and got ASPM working on both cards.

Here’s some info for people planning power usage in their low idle power servers.

Specs of my cards.
model: MCX4121A-ACAT
PSID: MT_2420110034
board revision: AB
Manufacturing date: 2017-04-23

Newest firmware version I installed
Firmware version 14.32.1010

I took some measurements of my Asrock n100dc-itx with and without this card.
Power measuring device: Shelly plug S
Power measured at the wall
Power adapter: some 2nd hand fujitsu 19v 90w
PC specs:
n100dc-itx + 32gb ram at 2400MTs + 512GB samsung PM981 + 1 Gb Ethernet link active

This testing setup has a limitation, n100dc-itx pcie slot has only 2 lanes of pcie 3.

Idle measurement with powertop --auto-tune: 7,10 W
Idle measurement with powertop --auto-tune + connectx-4 Lx no connection: ~13.12 W

Idle measurement with powertop --auto-tune + connectx-4 Lx + 1 SFP+ DAC cable and interface up: ~14,65 W

Idle measurement with powertop --auto-tune + connectx-4 Lx + 2 SFP+ DAC cable and interface up: ~15,46 W

Idle card adds 6,02W to this system
Idle card with 1 SFP+ connections adds 7,55W to this system
Idle card with 2 SFP+ connections adds 8,36W to this system

I don’t have sfp28 cables to test power usage at faster link speeds. Maybe I’ll revisit this at some point.

I’m not sure how much of the power usage delta comes from the card itself and not SOC pcie and other stuff. Card is not very hot at idle, I haven’t stress tested it because it would be pointless in this motherboard. (and hard to isolate NIC power usage during iperf)

Powertop shows that board reaches package C3 power state. The same result with and without the card.

lspci ASPM info

lspci -s 01:00.0 -vv |grep -i aspm
		LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
			ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
		LnkCtl:	ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
9 Likes
2

Thank you this is extremely helpful! Do you mind sharing the exact vendor link you bought your cards from so I can make sure I get the exact same ones? As recently as a few months ago people in this reddit thread are complaining of being unable to get ASPM working on their cards: https://www.reddit.com/r/homelab/comments/15eqm9s/mellanox_connectx4_questions_before_buying/

I don’t want to end up in that situation.

3

I’ve ordered this item 2 times and got 2 good cards from this seller. First card looked very generic. Second time it had IASER serial number tag, it was probably from Inspur branded device.
Their supply may be mixed.

https://aliexpress.com/item/1005005048786017.html

I’ve bough third one but I’m still waiting for the package. I can post results once it arrives.

1 Like
4

My third card arrived. Exactly the same model.

It came with firmware 14.18.1204 I’ve updated it to 14.23.1010
and card started to report L1 as available. The same as my previous cards.

LnkCap:	Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
5

Interesting. I just received what looks to be a genuine card from an ebay seller.

The card came with an old firmware and accepted the latest update directly from nvidia without putting up any fight:

flint -d /dev/mst/mt4117_pciconf0 query
Image type:            FS3
FW Version:            14.24.1000
FW Release Date:       26.11.2018
Product Version:       14.24.1000
Rom Info:              type=UEFI version=14.17.11 cpu=AMD64,AARCH64
                       type=PXE version=3.5.603 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             98039b0300ad6554        4
Base MAC:              98039bad6554            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2420110034
Security Attributes:   N/A

flint -d /dev/mst/mt4117_pciconf0 -i fw-ConnectX4Lx-rel-14_32_1010-MCX4121A-ACA_Ax-UEFI-14.25.17-FlexBoot-3.6.502.bin burn

    Current FW version on flash:  14.24.1000
    New FW version:               14.32.1010

FSMST_INITIALIZE -   OK
Writing Boot image component -   OK
Restoring signature                     - OK
-I- To load new FW run mlxfwreset or reboot machine.

flint -d /dev/mst/mt4117_pciconf0 query
Image type:            FS3
FW Version:            14.32.1010
FW Release Date:       1.12.2021
Product Version:       14.32.1010
Rom Info:              type=UEFI version=14.25.17 cpu=AMD64,AARCH64
                       type=PXE version=3.6.502 cpu=AMD64
Description:           UID                GuidsNumber
Base GUID:             98039b0300ad6554        4
Base MAC:              98039bad6554            4
Image VSD:             N/A
Device VSD:            N/A
PSID:                  MT_2420110034
Security Attributes:   N/A

Unfortunately ASPM seems to be giving me trouble, I suspect it’s something to do with my motherboard.

lspci
01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]

lspci -vvv -s 01:00.1 | grep -i aspm
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

lspci -vvv -s 01:00.0 | grep -i aspm
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

The system still achieves C3 on a clean boot with no VM’s running and the card installed though so that’s good news.

Cogno_iKWTP2FNtF
Cogno_iKWTP2FNtF657×760 33.9 KB

1 Like
6

If LnkCap: shows ASPM support your card should be fine.
I had similar situation on sketchy chinese erying motherboard.
You could try different motherboard or play with settings in configuration menu.

7

I also have these same cards ordered from Ebay (CX4121A MCX4121A-ACAT Mellanox ConnectX-4 Lx 25GbE) How did you update the firmware? I have only linux command line available.
EDIT: was able to update firmware and now aspm is available. Power consumption on the other hand didnt go down at least not yet. I have 40w idle power consumption from the wall with ryzen 7900

8

Can you tell us how you did it ?

1 Like