All the various Mellanox ConnectX-3 SFP+ cards, #100DaysOfHomeLab

Networking Networking Hardware
1

I have a little Proxmox home lab going on and for my #100DaysOfHomeLab today I would like to better understand the various ConnectX-3 cards.

Is there a preferred one for home use? The boundaries are SFP+ and 10gig.

Already ordered one from eBay, an CX311A variant. I wanted a dual port card, but was not liking the prices and I was already browsing for days, so I just picked one to get stuff rolling.

I found an Nvidia spec document for these cards, and it gives details for

  • MCX311A-XCAT, 1 port
  • MXC312A-XCBT, 2 port
  • MCX312B-XCBT, 2 port

The first one is straight forward, single port. How does the other two differ from each other? I can see they have different dimensions and use different components, but is one better than the other?

I also see some different model numbers, like CX322A, which google tells me is a Dell branded card. Are re-brands safe to buy? Or Dell will only work in Dell, Lenovo in Lenovo, etc.

I also saw ConnectX-2 cards being sold for a lot cheaper. I think those were made for PCIe 2.0. Is that something to consider for a home lab? My main concerns are power draw and heat generated, speed not so much.

2

My reaction to a Connectx-2 is that 109c was too hot. Not blaming the card per se, it was intended for a server chassis, but I didn’t like it. (technically a 80mm case fan cable tied to it will do enough - I presume however you don’t wish to burn down your home lab before the DR is in place).

I didn’t find the ConnectX-3 cards so expensive I considered any more of the 2’s. I bought various ConnectX-3 cards without a care to brand.

You may find SFP+ modules can be brand particular though this is typically by egregiously overpriced switches, for example. However I use generic ones with these cards (on which subject, don’t overlook DAC cables for <7m lengths, these are much cheaper than the 2 fiber modules).

3

I’ve installed the single port card last night into my PC, with a 10gig rj45 SFP+ adapter, and according to the mellanox firmware tool the card is 70C while idle.

I’m gonna use it with a DAC cable, so hopefully that alone will make it run cooler when it’s in the server, but now I’m thinking I’ll buy a small fan and design and 3D print some sort shroud and mount it on the heatsink. Would be good practice.