What Mellanox Card should I get???

Networking Networking Hardware
,
1

Hello everyone! This is my first post in these forums. I posted this same question in some subreddits and, unfortunately, I did get anywhere.
My main question concerns power consumption and c-states of Mellanox CoonnectX4 network cards. I am going to do a rebuild of my current server with power consumption in mind. I will be using a Z690 Gigabyte MoB and running TrueNas scale.
I was looking at getting either a 25G or 10G card, something like the MCX4121A-ACAT. However, I have been reading that many of these cards do not let the system enter proper c-states—forcing the system to consume way more power. So, I thought I would post this here and see what input you guys have, what model of card I should get and so on…

2

I have both MCX354-FCBT and MCX4121-ACUT, both doesn’t allow the CPU to go deeper than C2 state while running ESXi on balanced power profile.

1 Like
3

How many extra watts does that translate to on your system, roughly?

1 Like
4

I’m interested in this topic as well. I have a few E810 cards … and “nobody ever got fired for buying Mellanox”

1 Like
5

i don’t have a baseline measurement without NICs, but with the CX3 installed, the machine pulls about 50W from the wall at ~4% load in esxi7. I haven’t measured it with the cx4 installed but I’d expect it to be roughly the same.

specs of the machine
i5 12500
MSI Z690-A PRO DDR4
2* 32GB DDR4-3200 CL22 JEDEC
500W 80+ Silver PSU
2* SATA SSD
1* NVMe SSD
Add in NIC(cx3/cx4)

1 Like
6

If you want a 10gbe card why not consider intel out of curiosity?

7

Shameless plug my good experiences with MCX4121A-ACATs

I’d go with connectx-4 Lx over connectx-3.
Prices are similar, you get more speed (with backward compatibility) and a chance of ASPM depending of your luck and source of your card.

intel x520-da2 is a decent choice but afaik it doesn’t have ASPM.

1 Like
8

For some reason Linux doesn’t set ASPM properly according to the capabilities of my Mellanox cards (among others). It is possible to set these manually. It’s a little tedious to figure out, but can be scripted. However, in my case it did not yield sufficient benefits to keep it up.

2 Likes
9

Thanks for the input!

10

I’m not opposed to picking Intel. I’ve just heard that the Mellonox cards are good and cheap. I also know that they are good at negotiating down to 10G, so I could get a 25G card to future proof.

In you know of a good intel card that affordable and also good for power management, by all means let me know.

11

Try seeing if you can find these or their t1 and t2 variants in the L generation on ebay

1 Like
12

TBH I think Mellonox over Intel is best given the wide support and availability of used X4’s on ebay. X4’s also tend to be more power friendly than the Intel but not always.

Also, C-States… you turn off your NAS?

1 Like
13

Thanks for the input!
No - the NAS will be up 24/7.

Build will be an intel i5 12th gen
6 spinning disks in total (eventually 9)
An HBA and the nic.

I should probably invest in something that can read power from the wall.

14

Thanks for this! I found a vendor on eBay that has the same card and also seems to be a good seller. I just sent them a message asking what firmware was on the card. I am afraid I won’t have the skill set needed to flash and update myself - unless you are willing help?

( :

15

Read and internalise, it’s really straightforward.

1 Like