Hi level1techs forum, long time lurker first time poster. I’m slowly upgrading my homelab to 10 gigabit networking (currently point to point between three different systems). I mess around a fair bit with spark and ceph so I can definitely use and drive 10 to 25G with my current systems but I am also a cheapskate and not keen on dropping silly amounts of money for an expensive switch for mostly second hand servers. I know I can also buy second hand 10Gbe switches but a lot of them are still expensive, noisy (because they are designed for datacenters not homelabs) or very power hungry (I don’t really want to go over 100W idle and would prefer as low as possible). I’m also considering getting a Ubiquiti 16 XG or EnGenius Cloud ECS5512 but I am looking for cheaper options first.
So I thought, I’ve used a system as a lazy 10Gbe switch before (in an incredibly painful VMWare setup at work). It mostly worked and while it wasn’t getting amazing performance (it was fine for two machines but would start to chug on more than that). Some Xeon-d systems are starting to get end of lifed and appearing on ebay for not too much coin. Would something like a Supermicro SYS-E300-8D make a perfectly cromulent or would it introduce too much latency or overhead be worth buying specially for that purpose?
I am seeing mixed messages, many saying it doesn’t work or can’t be done but also some on the pfsense and netgate forums saying it kinda works but performance may suck. So the answers are kind of all over the place. Has anyone been crazy enough to try it for a small number of ports?
The takeaway after looking at a lot of information was that you could get away with it on a limited number of gigabit ports, but no matter what, you’re going to hit CPU bottleneck on 10G forwarding. And as far as I could tell, nothing like a switch asic on a pcie card exists.
Afaik, Mikrotik runs a custom linux on their switches. There are many expensive commercial Linux/BSDs that run on datacenter switches. ONL runs on some ONIE switches but is abandonware and SONiC runs on some ONIE switches but is new and has low adoption so far.
Ubiquiti Edgemax switches are Broadcom Fastpath under the hood. It’s fine, but difficult to manipulate with scripts. You have to use Expect which is difficult to write and can be unreliable.
Depends, on the the pci-e configuration and drivers and whether you’re firewalling, or not and if so, how. Whether you have any special fast forwarding options…and so on.
Damn, where were you with this video last week? /s
I wonder if VyOS is specifically doing things to leverage those chipset features or if vanilla Debian would have similar bridge performance. I like VyOS a lot, but you have to pay (considerably) for the stable release.
This system is really compelling if it can actually approach line speed bridging.
That is exactly what I was looking for @risk . I wonder, given that the video is from 2016 how much has improved on the software side since then. That certainly looks doable for my use case (a bunch of mixed 10/5/2.5 cards to connect together instead of currently point to point). A Xeon-d box should also do the trick as most of those boxes are designed as network appliances. I’ll report back with how I go because even if it fails I am after a new edge router anyway.
@oO.o, as i mentioned in a reply on your thread where you asked for 8x10G + some 1Gbps sprinkled for good measure
You’ll run out of CPU with 64byte packets, but at 1k or especially at 9k/16k if this is for storage, you’ll be fine - might even get away with routing and firewalling.
No. It’s a generic Linux kernel software bridge. Debian today would have the same performance if configured the same way.
There’s been improvements to bridge performance since (less copying and reassembly needed).
stp doesn’t really affect performance. 802.1x does (depends on the setup, you’d be using AES-NI on a xeon for it, bouncing the data across CPUs - I think you may need a skylake for 8x10Gbps because of ram bandwidth and L3 cache mesh).
VLANs affect performance, but as long as you use bridge filtering and not VLAN logical interfaces it’s negligible.
Do you know if there’s any way to get it to just push datagrams and avoid L3 altogether? I assume that would save some overhead?
What uses 64byte packets? The 10G in my case would storage plus vm networks coming out of hypervisors, so mixed 1500/9000mtu.
That’s good. I like VyOS a lot, but rolling release on the important network hardware is a no go for me.
I imagine this is why VyOS was outperforming pfsense (and presumably OpenBSD). I had dreamed of consolidating the whole gateway and switch but I do want to use pf, so probably still need to keep them separate.
vlan tagging can be offloaded on the NIC (at least some NICs). I’m sure pushing traffic between them would add load, but you think having them at all would?
I’m also just realizing that I don’t know how to enable/disable flow control on an interface. Seems like something I should have done by now…
Most of the L3 stack will ignore packets with other hosts mac addresses - various protocol handler code just doesn’t get called at all and it all stays in the bridge.
If it’s internet facing and you plan to (or want to for some reason) do your own ddos mitigation - it’s good to plan for 64B frames and not rely on closed source vendor provided code or ASICs.
Regular ISPs plan for 512B (e.g. 2Mpps translates to 1GBps), but with 1500 or 9000 byte frames it should be easy.
I haven’t watched the video recently, I assume he tried pfsense at first and it didn’t work out. flowtable is a more recent invention than that video and it’s mostly useful for L3 stuff. I suspect any performance difference back in 2016 was due to just better drivers. Unlike linux, BSD has a lot less interest when it comes to enterprise network hardware.
Yep, Linux bridges and most Linux ethernet drivers support it (e.g. the mellanox ones), but it doesn’t offload some of the bridging logic. … mellanox cards support offloading some of the bridging logic too, but that wasn’t used in that video (you can clearly see the cpu usage spiking up to quarter of a core when doing one of the tests where frames come in on one port of a nic and exit out the other at 10gbps between two hosts).
As above there are switches that run linux but that doesn’t mean linux is doing the work. Most commercial switches use ASICs to do the switching work, linux just manages the hardware, provides the configuration interface and does layer 3 routing, etc.
If you’re wanting to do 10 gig I’d suggest getting an actual switch.
It really depends ho many ports and how throughput/latency sensitive you are but I’d not recommend it.
For only a couple of machines communicating at once it looks doable at ~9Gbit/s which fits most of my homelab uses in the short term. I am also leaning towards just buying a couple of 40Gbit cards (I got one for $50 on ebay) and using the 10Gbit splitter DACs. This seems like an even cheaper way to do 10GBit with slightly lower power usage than the RJ45 cards. Latency isn’t the biggest concern for most of my use cases, raw bandwidth is more important and there is a parallel 1Gbit network.