Are you sure that’s a network issue and not smb? Possibly rdma is working on Linux but not on windows?
That is actually an Intel specific issue on Windows, other cards like Mellanox/Nvidia and Chelsio the 25gb link for their cards comes up instantly. The Intel one has some funky FEC issues. I have tried some e810-xxv first and ended up sending them back. Now I have both a CX-6 and Chelsio T6225-CR (cause I wanted to compare after the problem with Intel) and both cards actually work perfectly with no link issues like the Intel had.
Windows low network transfer speeds is also usually an RDMA issue, since Microsoft chose to lock the feature away behind licensing. You must upgrade to a “for Workstation” edition of Windows or a server edition to have RDMA in Windows. Stupid, I know.
Are you going direct NIC to NIC for your RoCEv2 connection? Or did you find a cheap switch that supports it? Typically I see people use iWARP instead because it works over normal switches and so is drastically cheaper and easier to use. Id be curious if you found a switch cause last I looked the only switches that supported the necessary tech were enterprise ones that were crazy expensive and very loud and power hungry.
3 Likes
Well I’m only getting ~ 8.3Gbits/sec according to iPerf. I think maybe something is mis-configured in my ESXi network setup on the server-side. The vmxnet3 adapter is only showing 10Gb within the VM, but I thought it could actually exceed that rate.
My ESX host is one of those lil Supermicro boxes w/ the integrated e810 SFP28 ports. I could have sworn I ran some tests earlier and was getting near 25Gb/sec speeds.
When I run iperf against another bare-metal machine on my network w/ another similar mellanox card, it gets a pretty consistent ~ 9.75 Gb/sec rate.
Blarg, back to the drawing board.
1 Like
I didn’t know I needed Windows for Workstation for RDMA. That might be my issue as well.
@Sawtaytoes, you might want to try Windows Server 2022 instead of Windows for Workstation for RDMA. You might have fewer compatibility issues by running Windows Server 2022 instead of Windows Workstation. I am currently testing a VM of Windows Server 2022. I have up to 180 days to try Windows Server before I pay for it. I have posted a link to download Windows Server 2022 in case you want to try it.
I’m on Windows for my desktop PC though. I don’t wanna run a server OS just so I can transfer files back and forth to my NAS at 25Gb.
Maybe I’m misunderstanding. Are you saying only the server needs to support RDMA? Does Linux not support it? I’m using TrueNAS SCALE.
1 Like
To better answer @Sawtaytoes’s question, I will explain why @Sawtaytoes might be having issues. Microsoft has removed RDMA from all versions of Windows except Windows for Workstation and Windows Server. As far as I know, no Linux distribution has put RDMA behind a paywall. I suggested Windows Server to @Sawtaytoes because Windows for Workstation is the most stripped-down version of Windows. He probably will have fewer issues if he uses Windows Server for all his Windows machines. I advise creating a Windows Server VM to act as his Windows system and connect it to his network. He will be able to use Windows Server for 180 days for free.
Also, if @Sawtaytoes isn’t directly connecting his TrueNas Scale system and his Windows system, he might need a special switch that can handle RDMA traffic.
2 Likes
Is the UniFi Pro Aggregation switch something that supports RDMA? How would I know? That’s the switch I have in between these two machines.
Looks like it’s not RDMA-capable. Is it a Windows issue where I need to upgrade to Windows for Workgroups?
I should have multichannel enabled as well, but does that not affect anything when I can’t max out a single NIC?
1 Like
No, definitely not. At least as far as “RDMA techs” in switches go. Typically you want something that has Datacenter Bridging (DCB functions) functionality in it, which are usually expensive switches. Mikrotik is the only brand I am aware of that has those features in “non expensive” switches, which by that I mean around $500-600. Other brands to get the functionality are $2000+. Switches with DCB feature set are required for all RDMA transfers besides iWARP type, and require hardware level support within the switch chip.
Switches without the full suite of features for more advanced RDMA transfers are still capable of iWARP RDMA though, which uses RDMA functionality in standard TCP packets, so any switch can do it.
SMB Direct is iWARP functionality via SMB protocol. It requires a Server OS or a For Workstation OS to use in Windows.
SMB Direct should automatically get used on an OS that supports it when using a NIC that supports it on both ends. If for some reason RDMA is not enabled by default on the NIC, you can enable it on NICs that are capable of the feature with the command:
Enable-NetAdapterRdma -Name "*"
Without SMB Direct you are typically limited to closer to 10gb transfers in Windows with SMB (often requiring jumbo frames to hit even that). With SMB Direct you can hit closer to 100gbps transfers.
4 Likes
Interesting. I had no clue RDMA was so specific, but also that it was so limiting to top-end speeds on these links.
Here’s an updated screenshot after enabling RDMA:
- It now says my ConnectX-6 is “Operational=True”. What’s that mean?
- It says my ConnectX-6 cards aren’t capable of supporting RDMA. Is that correct or do I need to update some drivers?
- If my switch doesn’t support it, I could add an LC coupler and avoid the switch entirely. I could leave the 10Gb copper link for other devices.
- The fastest speeds I’ve gotten are 16-18Gbps with these cards using iperf3 to Windows (apparently, this is not a valid test). I always thought I was doing something wrong. Why would you have an SFP28 slot if your switch doesn’t support those speeds? Or can it transfer at those speeds, just not for SMB?
- Are there any UniFi switches that support RDMA or would I have to buy a compatible Mickotik model if I want it part of the switch?
- What 25Gb NICs can I buy that support RDMA?
1 Like
What Windows are you running?
Windows 11 Pro 64-bit.
That OS doesnt have a license to activate RDMA.
1 Like
That’s new, thanks for mentioning it, I thought RDMA wouldn’t work with the CRS510.
I already have two C5 MCX516A-CDAT and am now thinking about getting C6 MCX631102AN-ADAT nics for my remaining systems. Can I expect 100Gb RDMA with the two C5 nics via the CRS510 or is the switch not suitable for that?
CRS510-8XS-2XQ Is on the Mikrotik compatible list:
https://help.mikrotik.com/docs/spaces/ROS/pages/189497483/Quality+of+Service#QualityofService-device
I believe the only 100gb Mikrotik switch that is NOT on the compatible list is the CRS520-4XS-16XQ-RM, which uses an older gen switch chip and not the 98DX series that has the DCB functionality.
Any of those switches with checkmarks for ECN and has PFC profiles should be compatible with RDMA features, though Im guessing the switches that have “Port/Queue Usage Stats” listed as “unreliable” may have some caveats attached to getting RDMA working well.
Check down farther in the page here for setup:
https://help.mikrotik.com/docs/spaces/ROS/pages/189497483/Quality+of+Service#QualityofService-RDMAoverConvergedEthernet(RoCE)
The MCX516A-CDAT cards are also the special PCIE 4.0 x16 versions! A rarity. Those will let you reach the full 100gb as you wont be limited by the PCIE slot bandwidth. The MCX631102AN-ADAT is an x8 card, so it can saturate 1 port at full 100gb but not both.
1 Like
yes that’s why I took these cards, thank you for taking the time to answer!
Guess I’m stuck with the slow speed then 
.
https://www.microsoft.com/en-us/d/windows-11-pro-for-workstations/dg7gmgf0kr4m
You can “upgrade” in place without reinstalling. Go to System → Activation. Change product key to the generic one: DXG7C-N36C4-C4HTG-X4T3X-2YV77. Restart your PC, then go back to that area and click Activate, enter the new For Workstation product key you bought. Then enable RDMA.
1 Like
For $300 where I can’t return it, I have some questions:
- Will this allow RDMA if I directly connect to the fiber jack of my NAS which is running TrueNAS?
- Does Linux (Debian) support RDMA out of the box or do I have to modify TrueNAS in some way?
- Since it sounds like my switch doesn’t support RDMA, which Microtik ones support it?
Looks like TrueNAS does not support RDMA out of the box either, though you can do some custom workarounds. Only when TrueNAS scale v25 is released next year will it have RDMA support built in, and only over NFS shares. Looks like Samba (the open source implementation of SMB protocol in Linux) does not fully support RDMA features which is why TrueNAS doesnt support it, and when they support it next year are only going to do so via NFS shares.
Edit: Actually, doing more research on it Samba does support RDMA. So I dont know why people over on TrueNAS forums are saying it doesnt, unless their own implementation version is not anywhere up to date with current Samba release.
So I guess that license upgrade would be worth it. Your server end doesnt have support for the feature set yet.
But Windows “For Workstation” itself does support RDMA, so if your NAS server was Windows based you could use it.
1 Like