This thread was started in Sep 2022.
I deployed my 100 Gbps IB network, in the basement of my house, starting in Dec 2018 (with the Mellanox ConnectX-4 NICs and cables) and then quickly realising that to be able to connect 4-6 nodes together, I needed a switch, which I bought in Feb 2019 and i’ve been using 100 Gbps IB ever since.
In terms of absolute cost, yes, it is more than 10GbE.
But in terms of $/Gbps, it actually ends up being cheaper than 10GbE.
So I jumped from 1 GbE straight to 100 Gbps IB (for my systems that can support it), skipping over 2.5G, 5G, 10G, 25G, 40G, and 50G.
Only my HPC applications can use anywhere CLOSE to 100 Gbps. (Usually tops out somewhere between 80-90 Gbps in reality.)
For “basic” storage and networking, (as my main storage server uses all HDDs, no SSDs), it is rare that I even break past 2.5G.
(However, I did play around with live migrating VMs using said 100 Gbps network, and depending on how you set it up, I was able to achieve > 100 Gbps data transfer rates when I was playing around with said live migrations.)
Thus, you CAN go faster, but for the value to be realised, the rest of the entire stack needs to be able to actually take advantage of it.
For me, I bought it for my HPC applications.
So, storage and “basic” networking use cases was actually more of a “fringe” benefit for me than its primary purpose.
(It’s primary purpose was for solving distributed sparse matrices (e.g. FEA) where I have topped out somewhere around 87-88 Gbps.)
If you want to use higher speed networking for storage, then it takes quite a bit to make sure that all of the stars and moon align together, for the link/physical speeds to be realised.
(And because my Mellanox ConnectX-4 cards have dual VPI ports, I did play with setting the second port LINK_TYPE from 1 (IB) to 2 (ETH), and then installed Proxmox, created a Linux network bridge (which I didn’t know this at the time, but it’s an ethernet bridge, which some of you might already know), and then tried to push data through it using iperf and was able to get ~23 Gbps (out of 25 Gbps per QSFP28 link (because 100 Gbps IB is actually 4 lanes of IB EDR (25 Gbps)), but that also meant that the Linux (ethernet) network bridge didn’t know/realise that it’s actually four 25 Gbps links that are basically “bonded” together and I didn’t have the knowledge nor the wherewithall to try to bond the individual lanes together.)
Host to host over the ETH link, I can get around 96.9 Gbps using 8 parallel streams going from Proxmox host to Proxmox host.
But with the network bridge, and going VM ↔ VM, it could only muster 23 Gbps (still tested with 8 parallel iperf streams).