FOSS Switch OS?

Networking
,
1

As I migrate away from Ubiquiti, I need to look at switch options. From working with the Ubiquiti routers I know there is hardware where you have a full OS that controls a switch that is build into the main board (on some of their routers, not their switches). I believe Mikrotik does this as well (usually MIPS processors in both cases I believe). Is there an OEM for such boards? It would be great to have a 48-port switch that runs linux or bsd on a small embedded system instead of broadcom fastpath, ios or whatever other proprietary platforms you usually find in switches.

OR, how feasible is simply bridging a bunch of NICs and using that as a switch? I assume some latency would be involved compared to purpose-built switch hardware, but not sure how much (assuming not doing fancy things like packet filtering, just forwarding packets datagrams).

Maybe something like this…

https://www.alibaba.com/product-detail/1U-High-Performance-x86-Network-Appliance_60669409174.html?spm=a2700.galleryofferlist.normal_offer.d_title.15a536bcTdqpiG

In which case the question would be, does anyone make a switch asic on a pcie card?

1 Like
2

Have you considered just buying the pfsense 1U rack mount option? It’s like $1100 but is low power and does 10GBaseT.

3

Cumulus has a compatibility list so I imagine something on there would work, not sure what some of those would set you back though. Network hardware compatibility list (HCL)

4

That’s kind of what I’m asking. Can I just get a server with a lot of network ports, bridge them with no filtering and still have good enough latency for things like 10Gb iscsi?

5

I think I looked at that before and it was mega $$ right? Or am I thinking of something else. Definitely want to keep things as FOSS as possible.

6

I more so meant the hardware listed and not Cumulus itself

7

You’ll never match the overall throughput of purpose-built switching hardware. With 48-ports of 10GbE ports you’re talking up to 960Gbps that all has to be routed in software.

Any “switch” you have in there will make those ports “dumb”. You can’t control the VLANs differently on each port, for example, like a smart switch, you might as well go dumb all the way.

The great white hope for eliminating proprietary OSes on switches is OpenFlow. There you never log-in to the device, you just have a program running on a computer telling it what to do.

1 Like
8

See you can do this on the Ubiquiti routers with the built-in switches. Iirc, this is also the case for many pieces of hardware that support dwrt. You can also find hardware accelerated packet forwarding and vlan tagging on router hardware. That’s what I’m looking for but not sure if you can find it in barebones form where I can load openbsd on it or if openbsd would even support the hardware features I’ve described if they are present.

9

I found something…

https://man.openbsd.org/switch.4

The switch driver provides a network interface pseudo-device. The interface is the forwarding part of a network switch and it works along with a controller, usually switchd(8) or switchctl(8), through the device /dev/switchN. The switch device and the controller communicate using the OpenFlow 1.3 protocol.

Found via:

Further investigation:

This is what I was referring to in the Ubiquiti routers:

I am mostly interested in the ip forwarding, vlan and bridge hardware offloading. I’m not entirely sure what bridge and forwarding offloading is under the hood. Are they talking about the CRC offloading or is it something else?

According to the em man page, OpenBSD does support hardware vlan tagging and crc, so maybe that would bring a layer 2 bridge/switch closer to line speed, especially within a single NIC? Idk though. So far I’ve only found discussion on why OpenBSD doesn’t support ToE, but that’s not really what I’m interested in.

I’m going to merge this out into it’s own topic…

1 Like
10

More interesting things:

Bridge driver in FreeBSD leverages better concurrency for 5x performance increase in FreeBSD13.

Hmm… VPP. Can’t find any mention of it in relation to OpenBSD, but I’m not categorically against using FreeBSD or Linux for this.

11

@freqlabs do you still haunt this place?

12

You are going to lose performance.

Operating system bridges are CPU bound. Switch bridges are handled by ASICS.

1 Like
13

@oO.o you may be interested to read up on Distributed Switch Architecture — The Linux Kernel documentation

Usually, switch asics allow for some forwarding and some transformation. Small/home switches you find in your will typically allow for VLANs, ipv4 nat up to a few thousand entries and often times pppoe encap/decap. Whatever they don’t recognize they’d forward to the cpu port. (Any icmp and so on). Enterprise/datacenter switch chips can do MPLS forwarding and ipv4/ipv6 forwarding. They’re more expensive, obviously, as they have more features use more silicone and power.

The datasheets/specs/data needed in order to implement drivers for either home or enterprise switch chips is scarce - not sure why.

1 Like
14

I know, but I don’t have a good idea of what the limit is or if there are any pcie cards that include said ASICS since there is some router hardware that does include switch ASICS on the board.

15

Ok, but what do you actually need? How many ports, how much bandwidth max per port, how much bandwidth in aggregate?

1 Like
16

Mikrotik will sell you the boards directly for projects like this.

2 Likes
17

I mean you could get a bunch of 4 port nics, set each nic to be a plan, and have like 6-8 in a box. Ta da?

Vlan

18

Up to 8 10GbE and around 20 1GbE which I realize is probably unrealistic but I still want to know what is realistic and what the most performing option is.

19

Of course I could. The question is about the performance of doing that.

20

I was looking at these or just buying a whole mikrotik switch and trying to put openbsd on it, but the switches have around 32/64MB of ram. Even with openbsd, idk if that’s usable. Other than that though, switch hardware with an OS on it is basically my ideal situation.

1 Like