Alternative 400gb/s switch for 4 dgx spark cluster?

Hardware Hub Networking
1

Hey forum, padawan here.

I’m trying to find the MikroTik CRS804-4DDQ-hRM switch in stock… anywhere… so I can set up my 4x DGX Spark founder’s edition cluster. The CRS812 seems to be completely sold out too. Both say backorders are expected to fill in November… but I have work to do untouchable from the larcenist hands of big AI and months is a lifetime away.

Anyone know a reputable place where I could get one?

Alternatively, anyone recommend another way to set this cluster up??

I’m non-technical trying to do technical things. It’s been a wonderful moving target for my curiosity but… just a lot. I’ll spare you all the long version of my journey and simply say - thank you.

Also to Wendell if you see this - you have my appreciation and gratitude

2

Just an idea.. Maybe you can set up a ring topology while waiting for the switch.

A quick google search showed me this; I don’t have any though but it seems possible:

1 Like
3

I mean, you’d have to bond some uplink, but Multilink[dot]us has some CRS812’s in stock :man_shrugging:

4

This is probably the best way to go; Run QSFP-DD DACs in a ring using dual-port NICs (or two NICs) and use a 3rd link (a 1/2.5/5/10GbE Ethernet) link for mgmt/internet/etc…I did a ring topo using 3x Dell XE7745 w/4x H200 GPU’s in a ring topo @ 400GbE using DAC’s and uplinked them with a 100GbE QSFP28 links to an HA pair of CRS520 ToR switch for data plane using an LACP bond0 bond w/9k jumbo frames, but all the token traffic is all backend vLLM between an OpenWebUI front-end in separate PVE cluster, so all user traffic for chats or, say, VSCode agents/etc terminates at the PVE cluster and LiteLLM API serves all the traffic between it and the front-end over the 100GbE uplinks and the cluster traffic does it’s thing in an isolated ring. That’s definitely the easiest/cheapest route. I do FRRouting and use OSPFv3 w/IPv6 along with a loopback interface that I inject my private v6 range with, peer them all up after distributing static IPs, and let FRR do it’s thing. Works great :+1: