40Gb/s ethernet on QSFP+ Mellanox ConnectX2?

Maybe I misunderstood something about Wendel’s comment on the Level1Linux video @ https://youtu.be/9ON2j7BxyF8?si=OclNG4UPjHzvbHrH&t=463 , but did Wendel actually say that 40Gb/s ethernet is possible on some cards that originally only supported 40Gb/s Infiniband? I tried searching more into this but couldn’t find any resources about this?

Yep, the ddn thread has more info. Cross flashing is a thing that works sometimes.

2 Likes

I assume you’re referring to [this]( DDNS Mellanox + SAS Combo Card Bootstrap Instructions (Linux) ) thread of yours? Additionally, is this original research on your part or did you see this concept mentioned elsewhere on the web? Your thorough thread appears to focus on the research did you, but not necessarily a set of instructions/guide as to how others can complete similar on other Mellanox ConnectX cards. Further, if you have actually managed to get some ConnectX cards to work with 40Gbe, I’d then imagine it may be feasible to add support for 2.5Gbe, 5Gbe, and 25Gbe, which these ConnectX cards traditionally did not support.

Yes, it’s not too obtuse arcana but no it won’t be possible to add more exotic support because it’s newer gen. 10/40 gigabit isn’t 10/25/50. It’s different series, symbols, etc.

Even the lane ordering on the ddn is “wrong” vs most mellanox phy

Distinction between features hardware supports but is gated off and new new features.

I learned the ddn cards hard can’t do 40gbe but I fixed the iommu problem on the sas side.

It’s psid and ini manipulation and flashing with the mstflint utility

2 Likes

I only [earlier] just-today learned a bit about .ini files, so I guess I’ll need to go digging a bit more.

Thanks for starting the research and conversation!

1 Like