I would like to connect two of NVIDIA’s Jetson AGXs with high speed Ethernet to minimise latency. I tried connecting with 1 Gbps link but the latency is too high for my needs. Would it help if I switched to 10 Gbps Ethernet instead? If yes, since AGX doesn’t have a 10Gbps interface, I would need to find a 10 Gbps network card with Gen3 PCIe that plays nice with Linux. Any recommendations?
I’ve had results I’m happy with from the mellanox connectx-3 to work with linux (&pfsense) - is available ebay, affordably and doesn’t overheat when not in a server chassis.
I’m not latency sensitive however.
Connect with a DAC cable and this won’t be more than 3 parts total, avoiding fibre/SFP (and it’s supposed to be lower latency than fibre)
The only thing I’d say against it is - I occasionally have to reset it after wake from sleep on one system often enough I use a script for it.
1 Like
I don’t know the Jetson AGXs that well but I’d assume that they mostly behave like normal server systems.
First of all probably all 10G+ NICs should work perfectly fine in Linux system. More than 90% of all servers (where you usually find those cards) run Linux. Therefore it’s usually way easier to get in running than on other OS.
I’d recommend you try to specify what you need or rather benefit from.
I would definitely choose Infiniband over Ethernet for at least two reasons: Infiniband always comes with a TCPoE and RDMA. Both drastically reduce latency and improve throughput at given bandwidth.
TCPoE stands for TCP offload Engine which is basically a processor that is purposely built to reduce CPU overhead of TCP pakets (At 10Gb/s+ TCP stuff causes a significant CPU overhead). RDMA allows data transfers with fewer copies of data which improves latencys and throughput. Infiniband NICs aren’t that common usually you find IBoE (=InfiniBand over Ethernet) cards which are totally fine. Just look for “Infiniband” or “IBoE” in the data sheet.
NICs with the Marvell/Aquantia AQtion AQC107 chipset (found on those $100USD 10Gb NICs like the ASUS XG-C100C) can do neither.
Intels x550 chipsets are way better and do have some offload capabilities but lack RDMA support.
From my experience I can recommend the Mellanox connectx NICs.
There are good tutorials that show how to enable IBoE for those cards.
If all you do is point to point communication and used hardware is an option I’d rather go with 40Gb QSFP+ cards because you don’t need a switch. At least here in germany you can get used 40Gb Mellanox connectx-3 NICs for around $100 USD including taxes and two years warranty.
@exovert @luukas Thanks for the advice! I bought the ConnectX-3 since you guys recommended it and inifiband in general. Unfortunately I have run into issues due to my limited knowledge. The driver installation on Jetson AGX was a breeze, but I realize that I cannot connect the 10 Gig ethernet cables I bought directly into the ConnextX-3.
Could you please tell me which cables I should connect into the tall slot? Since Mellanox has discontinued this product, their website doesn’t have much information.
Second, would it be possible to combine some transceivers and convert the infiniband signal into an ethernet signal on the other side? I want the NIC to connect to another device which does not have an infiniband port, neither a PCI slot. I don’t necessarily need the 10G bandwidth, so I am ready to take some performance hit as long as the other device can connect to my AGX.
I’m unsure which variant of card(s) you’ve acquired (it could have SFP+, or QSFP), and if I’ve not misread the application seems to differ a little from above, of connecting two Jetsons with the cards. I’d want to be sure what you have now and need to get to.
Some background reference on these connectors
The rectangle connector on the ConnectX-3 is a SFP+ (or QSFP) - which accepts an SFP+ module for the media you need to use (DAC, RJ45 Cat 6/7, fibre).
RJ45 SFP+ modules are not too cheap, where I am at least, but that might get you to RJ45 <> RJ45 each end, assuming you have the SFP+ variant.
Works with all 3 major OS’. Good Luck with your project.
Which specific connectx-3 do you have? MCX354A-FCBT ? … one of the other ones? One of the IBM or HP branded clones?
What do you have on the other end, RJ45, some kind of sfp-ish? What cables were you hoping to use?
@risk I have the MCX311A-XCAT variant. The other device has a standard RJ45 connector. I was hoping to use a Cat 6 or 7 cable (I’m open to buying other ethernet cables).
Oh, well then that makes things easier - that’s SFP+, you just need an SFP+ 10GBASE-T module.
I use FiberStore (FS.com) for all of my networking stuff that I don’t trust eBay for:
None of the Mellanox ConnectX-n products I’ve used have had a vendor lock for SFP transceivers (Cisco, Juniper, and a few others like to require matching branded SFP transceivers) and the ones FS sells flashed as ‘generic’ have always worked. The ‘manufacturer’ selection they have is just the ID string that’s flashed to an EEPROM on the module.
If you want to go the eBay route and get something used, basically any 10GBASE-T SFP+ RJ45 module should work. They’re everywhere.
FS also has everything up through 20ft CAT8 patch cables for like $11. I love them lol
2 Likes