Ahh…that makes sense.
No, it’s not.
Remote direct memory access (RDMA) over converged Ethernet is different than Infiniband.
So here is how you can check:
If you have the “Infiniband Support” group of tools installed on your OS, then you should have a tool called mstconfig.
If you run lspci | grep Mellanox, you should see something like:
04:00.0 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
04:00.1 Infiniband controller: Mellanox Technologies MT27700 Family [ConnectX-4]
The 04:00.0 is going to be the device ID that you are going to use for the mstconfig command:
sudo mstconfig -d 04:00.0 q
and it should output something like this:
Device #1:
----------
Device type: ConnectX4
Name: N/A
Description: N/A
Device: 04:00.0
Configurations: Next Boot
MEMIC_BAR_SIZE 0
MEMIC_SIZE_LIMIT _256KB(1)
FLEX_PARSER_PROFILE_ENABLE 0
FLEX_IPV4_OVER_VXLAN_PORT 0
ROCE_NEXT_PROTOCOL 254
NON_PREFETCHABLE_PF_BAR False(0)
NUM_OF_VFS 8
FPP_EN True(1)
SRIOV_EN False(0)
PF_LOG_BAR_SIZE 5
VF_LOG_BAR_SIZE 1
NUM_PF_MSIX 63
NUM_VF_MSIX 11
INT_LOG_MAX_PAYLOAD_SIZE AUTOMATIC(0)
SW_RECOVERY_ON_ERRORS False(0)
RESET_WITH_HOST_ON_ERRORS False(0)
CQE_COMPRESSION BALANCED(0)
IP_OVER_VXLAN_EN False(0)
PCI_ATOMIC_MODE PCI_ATOMIC_DISABLED_EXT_ATOMIC_ENABLED(0)
LRO_LOG_TIMEOUT0 6
LRO_LOG_TIMEOUT1 7
LRO_LOG_TIMEOUT2 8
LRO_LOG_TIMEOUT3 13
LOG_DCR_HASH_TABLE_SIZE 14
DCR_LIFO_SIZE 16384
LINK_TYPE_P1 IB(1)
LINK_TYPE_P2 IB(1)
ROCE_CC_PRIO_MASK_P1 255
ROCE_CC_ALGORITHM_P1 ECN(0)
ROCE_CC_PRIO_MASK_P2 255
ROCE_CC_ALGORITHM_P2 ECN(0)
CLAMP_TGT_RATE_AFTER_TIME_INC_P1 True(1)
CLAMP_TGT_RATE_P1 False(0)
RPG_TIME_RESET_P1 300
RPG_BYTE_RESET_P1 32767
RPG_THRESHOLD_P1 1
RPG_MAX_RATE_P1 0
RPG_AI_RATE_P1 5
RPG_HAI_RATE_P1 50
RPG_GD_P1 11
RPG_MIN_DEC_FAC_P1 50
RPG_MIN_RATE_P1 1
RATE_TO_SET_ON_FIRST_CNP_P1 0
DCE_TCP_G_P1 1019
DCE_TCP_RTT_P1 1
RATE_REDUCE_MONITOR_PERIOD_P1 4
INITIAL_ALPHA_VALUE_P1 1023
MIN_TIME_BETWEEN_CNPS_P1 0
CNP_802P_PRIO_P1 6
CNP_DSCP_P1 48
CLAMP_TGT_RATE_AFTER_TIME_INC_P2 True(1)
CLAMP_TGT_RATE_P2 False(0)
RPG_TIME_RESET_P2 300
RPG_BYTE_RESET_P2 32767
RPG_THRESHOLD_P2 1
RPG_MAX_RATE_P2 0
RPG_AI_RATE_P2 5
RPG_HAI_RATE_P2 50
RPG_GD_P2 11
RPG_MIN_DEC_FAC_P2 50
RPG_MIN_RATE_P2 1
RATE_TO_SET_ON_FIRST_CNP_P2 0
DCE_TCP_G_P2 1019
DCE_TCP_RTT_P2 1
RATE_REDUCE_MONITOR_PERIOD_P2 4
INITIAL_ALPHA_VALUE_P2 1023
MIN_TIME_BETWEEN_CNPS_P2 0
CNP_802P_PRIO_P2 6
CNP_DSCP_P2 48
LLDP_NB_DCBX_P1 False(0)
LLDP_NB_RX_MODE_P1 OFF(0)
LLDP_NB_TX_MODE_P1 OFF(0)
LLDP_NB_DCBX_P2 False(0)
LLDP_NB_RX_MODE_P2 OFF(0)
LLDP_NB_TX_MODE_P2 OFF(0)
DCBX_IEEE_P1 True(1)
DCBX_CEE_P1 True(1)
DCBX_WILLING_P1 True(1)
DCBX_IEEE_P2 True(1)
DCBX_CEE_P2 True(1)
DCBX_WILLING_P2 True(1)
KEEP_ETH_LINK_UP_P1 True(1)
KEEP_IB_LINK_UP_P1 False(0)
KEEP_LINK_UP_ON_BOOT_P1 False(0)
KEEP_LINK_UP_ON_STANDBY_P1 False(0)
KEEP_ETH_LINK_UP_P2 True(1)
KEEP_IB_LINK_UP_P2 False(0)
KEEP_LINK_UP_ON_BOOT_P2 False(0)
KEEP_LINK_UP_ON_STANDBY_P2 False(0)
NUM_OF_VL_P1 _4_VLs(3)
NUM_OF_TC_P1 _8_TCs(0)
NUM_OF_PFC_P1 8
NUM_OF_VL_P2 _4_VLs(3)
NUM_OF_TC_P2 _8_TCs(0)
NUM_OF_PFC_P2 8
DUP_MAC_ACTION_P1 LAST_CFG(0)
SRIOV_IB_ROUTING_MODE_P1 LID(1)
IB_ROUTING_MODE_P1 LID(1)
DUP_MAC_ACTION_P2 LAST_CFG(0)
SRIOV_IB_ROUTING_MODE_P2 LID(1)
IB_ROUTING_MODE_P2 LID(1)
PCI_WR_ORDERING per_mkey(0)
MULTI_PORT_VHCA_EN False(0)
PORT_OWNER True(1)
ALLOW_RD_COUNTERS True(1)
RENEG_ON_CHANGE True(1)
TRACER_ENABLE True(1)
BOOT_UNDI_NETWORK_WAIT 0
UEFI_HII_EN True(1)
BOOT_DBG_LOG False(0)
UEFI_LOGS DISABLED(0)
BOOT_VLAN 1
LEGACY_BOOT_PROTOCOL NONE(0)
BOOT_RETRY_CNT NONE(0)
BOOT_LACP_DIS True(1)
BOOT_VLAN_EN False(0)
BOOT_PKEY 0
ADVANCED_PCI_SETTINGS False(0)
SAFE_MODE_THRESHOLD 10
SAFE_MODE_ENABLE True(1)
You will notice that there is an entry in there for LINK_TYPE_P1 which, in my case, says “IB(1)”.
I forget if your card has Mellanox’s virtual port interface (VPI) technology where you can change the port type from Infiniband to Ethernet or Ethernet back to Infiniband.
One of the ways that you could test it would be to issue the command:
mstconfig -d 04:00.0 set LINK_TYPE_P1=2
and then query your card again to make sure that the changes were successful.
If it worked, then you are now running your Infiniband card with the ports in Ethernet mode.
If not, then this might be one of the reasons why your attempts at trying to run RDMA over Converged Ethernet (RoCE) isn’t working for you.
(For me, I am running straight RDMA (over Infiniband), and so, I don’t have this problem with the whole Ethernet layer/side of things.)
Your cards should be ETH only then, which in that case, you are absolutely correct in that you don’t need to have a subnet manager running at all then.
Yeah…so it really depends.
Like I am using four HGST 6 TB SATA 6 Gbps 7200 rpm mechanically rotating hard disk drives in RAID0, so it’s not terribly fast either. (The best that I’ve been able to get is about 800 MB/s combined write speed or about 200 MB/s contribution per drive, which works out to be around 6.4 Gbps.)
So by that metric, even regular 10 GbE would work and it would be a LOT easier to set up than RoCE.
In my use case which is understandably, a lot different than what most other people use it for is I actually will use the 100 Gbps Infiniband upto around 80 Gbps or so for high performance computing/computer aided engineering applications.
That’s where it really comes into play.
All of this NFS over RDMA stuff is just an added extra bonus of “things that I can do with 100 Gbps Infiniband” that I couldn’t really do with normal gigabit ethernet.
I think that when I first got my hardware, I did run some tests with straight NFS (no RDMA) and it was maybe doing about 150 MB/s writes.
So, needless to say, the RDMA DOES makes things faster, but with mechanically rotating hard disk drives, it has an upper limit based on the physics of the medium itself. (Not going to be breaking any records anytime soon.)
Conversely, because I went straight to 100 Gbps Infiniband, I’ve kind of skipped over the 10 Gigabit layer/step entirely (along with also skipping 25G, 40G, 50G, and even 56G).
So I use it as much as possible, in as many ways as possible, in as many places/systems as possible.