Hi,
I jumped on the LTT InfinBand band wagon to get to the promised lands of above Gigabit home-networking.
I'm not there jet, but this might help or entertain you, if you are interested in this topic.
Lets begin with the market, "Ebay".
Since Linus video, it has changed or atleast is a bit different here in Germany.
The Mellanox HCAs are nearly unavailable and very expensive in comparison.
He paid 25$, i'd have to pay like 80€ each. Probably even more.
So no Mellanox MHQH19B.
MHQH19 sometimes go for 40€ but that's still like 70% more than what Linus paid.
My choice since I'm mainly on Linux were his first pick, QLogic [QLE7340].(http://www.qlogic.com/OEMPartnerships/HuaweiSymantec/Documents/ds_7340QDR_HCA.pdf)
Paid 25€ each, and they are pretty broadly available.
Comparison QLogic to Mellanox.
QLogic HCAs are ASIC chips without firmware, unlike the Mellanox HCAs where the firmware can cause problems.
Drivers are the main difference.
Mellanox officially has very broad support. QLogic doesn't, or it atleast doesn't tell you.
QLogic / Intel uploaded the source code to Github.
qib
Drivers aka OFED
OFED is a Software by the Open Fabrics Alliance and it is available for a few Linux distros and Windows.
OFED as a package of many drivers, modules and programs.
What I did, no OFED for me!
I'm currently on Mint. Fired up Synaptics packagemanager, and searched for Infiiband, found and installed Infiniband-diags and dependencies.
Next is the subnet-manager named opensm who's also in the OFED package.
Found this guide and went with it.
I later learned that the qib driver is somehow already available so I tugged
qib # QLogic HCA driver
into the driver section of 4.4 of the guide. It should automatically use the qib driver even without the entry, but that's how I learned about qib being there.
Loaded all the modules
sudo modprobe modulename
except, i think "ib_sdp", that wasn't available.
is it working?
Look at 4.4 of the guide.
For me, it detected the cards, but the link-state was Down and the cards were sleeping.
The fu**en cable.
Bought a 10m Mellanox qsfp to qsfp copper cable for 20€.
The problem is that the HCAs just ignore it.
Link-state is always down, with or without plugged in cable. No blinken lights too.
It looks like incompatibility is possible. Something you can't read anywhere, at least i couldn't find anything searching for it.
Maybe it has to do with QSFP vs QSFP+ but my HCAs are only QSFP and the cables too.
Probably a conflickt reasoned by Mellanox vs QLogic.
Supported cables are mentioned in the QLE7340 data-sheet, they sadly are more expensive but available.
Ordered one just now.
QLogic and Infiniband on FreeNAS.
Freenas sadly doesn't support infiniband out of the box.
The FreeBSD kernel isn't compiled with the drivers and modules.
Opensource to the rescue:
freenas-infiniband
This currently only works with the Mellanox cards, but the qib driver is in FreeBSD since 2010 (i think) and probably just needs to be compiled into the kernel.
I'm going to try this, and if it works, you'll have it in the repo soon.
Please excuse the more or less random order, if you have and idea on how to improve it, please let me know, i'll update this post.
At this point, I want to thank, Justin Clift, the owner of the FreeNas-InfiniBand repo, who helped me figure out the qib driver story, I would have been lost without him!
That's it, i'll update this after i get the new cable and the new FreeNAS kernel, till then!
Thanks for reading!