Hello folks!
With the recent updates on Mikrotik RouterOS 7.15 they’ve introduced support to PFC and ECN which are vital to enable RDMA with RoCE.
I have a setup of 3x servers with a pair of Dual 25GbE Connect-X 4 each (which supports RDMA with RoCE v2). I’d like to use two ports (one on each NIC) on each server to connect to a CRS326-24S+2Q+ (which supports the new QoS features on the switch chip) and make them to use RDMA for the storage cluster traffic on vSAN.
All ports on the switch are part of a single HW bridge using VLAN filtering. The 6 ports for the storage are on its own VLAN.
I’ve read Mikrotik docs on the QoS but I couldn’t wrap my head around its concepts.
On a Cisco Nexus switch for example, this is the configuration I would use that works perfectly fine:
!Enable Data Center Bridging (extension of LLDP)
feature lldp
!qos config for RDMA
class-map type qos match-all ROCE_CLASS
match cos 3
match dscp 26
policy-map type qos ROCE_POLICY
class ROCE_CLASS
set qos-group 3
policy-map type queuing ROCE_QUEUING_OUT
class type queuing c-out-8q-q3
bandwidth remaining percent 50
random-detect minimum-threshold 150 kbytes maximum-threshold 3000 kbytes drop-probability 7 weight 0 ecn
class type queuing c-out-8q-q2
bandwidth percent 0
class type queuing c-out-8q-q1
bandwidth percent 0
class type queuing c-out-8q-q-default
bandwidth remaining percent 50
class-map type network-qos match-any ROCE_NETWORK_CLASS
match qos-group 3
policy-map type network-qos ROCE_NETWORK_POLICY
class type network-qos ROCE_NETWORK_CLASS
pause pfc-cos 3
mtu 9216
class type network-qos c-8q-nq-default
mtu 9216
system qos
service-policy type network-qos ROCE_NETWORK_POLICY
!Run on all interfaces used for vsan/management
int eth <X/X>
lldp dcbx version ieee
priority-flow-control mode on
priority-flow-control watch-dog-interval on
service-policy type qos input ROCE_POLICY
service-policy type queuing output ROCE_QUEUING_OUT
Do we have any Mikrotik folk(s) around that could shed a light on how to configure the MT switch to achieve the same PFC/ECN settings? (ignore the DCBX settings on the Cisco example as they are not supported on MT side)
Thank you!
I really appreciate any help.