10Gbps+ Lan Party *Sense Router/Server Build Suggestions/Recommendations

Networking Networking Hardware
1

Hello.
I’m building a *BSD (pfSense/OPNSense) router/server for a 200+ player Lan Party. And I need some suggestions from people that are a whole lot more in the know about such builds.

First some background:
Me and my team have organised 2 big (100-150 player) Lans in the last 2 years.
Both times we used a X99 build with a [email protected] as the router.
It also contained 3* 4x 1Gbps Intel NICs as links to the switches and a 2 port 10Gbps card from QLogic (HP 530T) as the main link in.
We never came even came close to maxing out the CPU but we have decided to replace the X99 build with something more server like as we are expecting more and more players in the near feature.

Requirements
We are looking to build a Router/Server that can handle at least 10Gbps and would stay with us for multiple years. That’s including caching stuff like Steam, Microsoft updates, …
The machine would also have a couple of Jails/VM’s for game servers.

The current budget is 3000€ (That’s for the motherboard, CPU and Ram only).
We plan on filling the machine up with:
2x 2 Port 10Gbps Ethernet cards*,
2x 2 Port 10Gbps Fiber cards*,
n* 4 Port 1Gbps Ethernet cards**.

I have currently found 2 motherboards that I assume would be good enough for our use case.

MW51-HP0 - Intel LGA2066 socket
I assume Intel CPUs are tried and tested in such a use case.
The CPUs boost to a higher clock then any AMD EPYC CPU on the market currently but due to the budget I would be trading cores, memory and PCIe lanes for addition clock speed.***
The fact that the motherboard uses a PLX chip to get some of the lanes also concerns me. I assume if I used 4x PCIe cards in the last 4 slots there would be no issue, but upgrading down the line…
Also the board only comes with 2x 1Gbps NICs (shared over DMI3).

MZ31-AR0 - AMD SP3 Socket
The board already has 2x SFP+ 10Gbps slots eliminating one PCIe card.
Everything important is directly connected to the CPU so no PLX chips.
Also a bunch more ram slots.
The problems I see are that AMD EPYC is clocked lower then Intel Chips*** and looking at the OpenBSD documentation AMD EPYC is not mentioned anywhere (and yes I know that the CPU works with OpenBSD looking at the Phoronix testing).

Why not use a dual socket server?
We tried using a dual socket Intel server but we had some problems, if one of the cards was attached to CPU1 and the rest to CPU0, they would sometimes drop packets or not even link at all.

Why not upgrade the X99 build?
The same issue as listed for the Intel platform above, PLX chips.

TL;DR:
More Cores (AMD) or More GHz (Intel) for a *Sense router build?

All recommendations are welcome. Just make sure they are reasonable

* The cards would be used for incoming links from the school or facility that we would host the party at. Some are only able to provide high speed Fiber other Ethernet.
** The cards would be upgraded as the need for more connectivity would grow. The reason we currently want multiple out links is due to most of our switches only support Gigabit. Also the need for multiple separate Lans.
*** Due to only testing with high clock speed CPUs, I don’t really know what pfSense/OPNSense would prefer, cores or speed. If anyone has done the testing and has some number to share, please let me know in the comments.

(I might edit this to add more information later so I reserve this small section for edit notes. Also IF this build becomes reality sooner then later I don’t mind posting performance numbers and a build log here.)

Edit 1: Add a note about recommendations.

2

with that much going on i wonder how much SQM would help prioritization of traffic and packet types

1 Like
3

We are planning on setting up a simple traffic sharper to limit the amount of bandwidth a user can use.
We did learn our lesson the first time when a user hogged all the bandwidth with torrents.

4

I’m not a networking expert by any means…but in the case of 200 people connecting to one server, seems like infrastructure could matter more than horsepower. You mentioned an incoming high speed fiber Ethernet? Do you connect the gaming PCs via copper? What sort of switch were you planning on using?

1 Like
5

Valid question.
We are upgrading the networking setup (switches) slowly but this was the latest setup:
Cisco 3850-48U-S as the main switch. 12 Gigabit connections going to it (LACP). Then from that switch we ran 4 LC fiber connection (so 2 switches) to the most remote switches, everything else had a 2x 1Gbps links going to it.
We are trying to make the connections are redundant as possible since people that are walking around could accidentally clip a cable or something else could happen to it.

This are all the switches I own and use (other switches are provided by other organizers or sponsors):
image

6

Random musings:

Give everyone 100mbps for free contention.
Use switches where you can control port speed. If someone is an ‘arse’ give them 10mbps.
Proper hardware router may be preferable to a software one.
Link speed is only part of the equation, pps is important too.

1 Like
7
  1. Already planing on doing that (Internet), but we still want to get 1Gbps (LAN) to every user.
  2. Limiting users via link speed on the switches would be a good idea, if we didn’t have caching. Also we need to limit how many connections can a user have/make.
  3. True. But hardware that could handle all of that would cost way to much in the start and the long run.
  4. Also true. That’s why we are going to limit the amount of connections a user can make. Even without that, I hardly believe that this will be an issue for this class of hardware.
8

bump

9

If I read all that correctly…

You have a bunch of switches (that have users connected) are redundantly hooked up to one core switch?

10

Correct.
But that’s the best way of doing it.
Otherwise the bridge (LAN) would be handled by the Router/Switch and that would be very very slow.

11

I guess you could also do a 2950x (higher clock, half lanes epyc)

Curious, why not Linux?

(You’re only doing connection tracking and htb, right?)

12

I would really love to pick up a TR platform.
But the isn’t a single motherboard with more then 4/5 PCIe slots.
(It’s possible to get 16+16+8+8+4+4+4 PCIe connections using M.2 to PCIe adapters, but that a jenky setup)

Why not Linux?
I don’t know. Really anything goes in my case.
But I must also teach others how to work and maintain the system when I’m not around.

If you know of a good Linux PfSense competitor let me know.

13

But you only need that because you can’t afford to make up your mind about upstream connectivity if I get you right?

e.g. A pair of quad port sfp+ (easily fits into a pair of 3.0 x16 slots) going to a couple of switches with some 10G ports would have enough connectivity? and you could get sfp+ to rj45 as needed for your Ethernet uplink runs. Or you could get TR motherboards with 10G copper.

There are switches like ubiquiti es-48-lite or us-48 (~ $300 - 450) that you can use to provide gigabit connectivity at the endpoints and those have 10G uplinks that you could connect to your 10G core where your router/services would live.

For $4k/$5k you could probably build this from scratch switches included; but you need a spreadsheet or two or more various stages of evolution for this if you want to stretch your current budget.

14

We are looking for more a upgradable system down the road.
Looking at TR it looks OK, but 1 slot goes out the windows since you need a GPU, so only 4 slots left (16+16+8+8 from the CPU).

But going with TR for the starting phase wouldn’t be a bad choice since we could always just sell the parts and upgrade the rest.

Also like you said the PCIe connectivity wouldn’t be that much of a issue.
We just got a 2x 40Gbps Infiniband card that can adapt to 2x 4x 10Gbps SFP+ so we could use that as a connection to the switches. Inbound would be handled by a single 2x 10Gbps card (fiber or RJ45) since there is no way to get faster internet at any location ATM.

For switches we got 16x WS-C3750-48TS-E and are planing to connect them via SFP OR RJ45 with adapters.
Don’t know which would really be cheaper at that point since we still need to get 10Gbps SFP+ to the “main” switch. Looking at something like 2x CRS328-4C-20S-4S+RM.

15

Just don’t put a GPU in the machine :slight_smile:

I thought about the 40gb breakouts, but I’ve never seen a dac cable that has qsfp+ module on one side, and sfp+ slots on the other side. Btw, what nic? (Mellanox or Intel?)
The only way around it I can think of, if you don’t want to collocate all the switches is to use as 10Gbps switch as a coupler/female-female adapter to get 10G sfp+ transceiver ports. E.g. a quanta lb6 or similar (24SFP+ for 250 if you’re lucky, very noisy, but maybe don’t care for a lan, $10 per port).

As for access switches, CSS326-24G-2S+RM are a good shape for 140-150, (24copper gig ports and 2xSFP+) get 8 of them, get some cheap premade multimode cables to hook them up into that quanta, and there you have your ~800Mbps non blocking + bunch of 10Gbps ports for cheap cache servers.

On the topic of quanta, you could also look at those old quanta lb4 switches, 48G ports and 2xSFP+ uplink , for same money as the CSS326, but unlike CSS326 that only has a web UI these have ssh. Less uplink bw, double the port count, not as much cross sectional bandwidth from one end of the lan to the other, but if you’re happy with 400Mbps under severe load (unlikely to get that load on a lan), these might save you money since you need half the 10G ports, and half the access switches.

Sell those 3750 to certificate chasing folk, they’d be grateful, you don’t really have a need for a bunch of 10Gigless switches do you?

16

Thanks for the info and thoughts.
Mellanox MT26428 card and this dac.

The Quanta LB6M is a far better option. But I don’t know if I’ll be able to get it for cheap, since EU. Also how are firmware updates on it? Since most Fiber Switches kill you with additional licences.

Also for rest of the switches, currently we don’t need (want) to upgrade them since there really is no need for Gigabit connectivity to every player.
It also makes it easier for us to split the tournament players (1G) and normal users (100M).

17

PCI-E 2.0 x8 ? That’s 40Gbps max … more like 30-32 in practice. And don’t forget to enable 4k pcie transaction size on your motherboard.

They suck, but which ones don’t?

LB6/LB6M is old and deprecated, it’s been old and deprecated for a number of years; it used to be used by Amazon and a whole bunch of other cloudy companies in their DCs ~10 years ago as a tor (top-of-rack switch), the market is now flooded with them since all of these have switched to at least dual 40G per host (and even that’s now getting recycled into tier 2 DCs with 100/200G being the new hotness), and they’re selling as quickly as they can be refurbished back to latest quanta factory firmware and retested. Lots of these companies build their own switch firmwares (sometimes you can land these for less than 200 with strange weird useless firmwares and/or 1 or 2 dead ports). Latest quanta firmwares are still ancient, they’re there, easy to get, but the switch is just a carrier board for old broadcom lots-o-pins switching chips that broadcom no longer makes, that have their own crappy driver sdk trying to be smart about IP layer 3 stuff. You’re getting pretty much the same experience a team of highly skilled people working on cluster networking at these big DCs got 10years ago when they decided it’s worth the hassle to negotiate the ability to build their own firmware for it, and sign a bunch of NDAs with broadcom directly.

On serve the home forums, there are multiple threads on lb6m, one with purchasing deals, one with discussion… In the super long discussion thread you’ll find a) hardware is the same as brocade turboiron 24x (yes I know brocade software has a history of suckiness and tickling snmp crashing the switch, but brocade typically provides support subscriptions to their customers) b) links to guides on how to flash it with that firmware c) links to brocade documentation d) guides on how to cross compile software for the powerpc architecture and ancient kernel that the switch cpu runs on, so you can run your own stuff. Most people just flash turboiron fw and forget it since it’s good enough - some vlans, a bit of L3, some IGMP and DHCP filtering, okayish ssh and you’re done.

18

Thank you for input.
The Mellanox card will probably never even see speeds like that, since we current don’t know of a location that could provide that kind of speed.

Do you maybe know how a Chelsio 10GB NIC would fair up to a Intel NIC?
We currently do have a Intel NIC and looking at the official Chelsio benchmarks the Chelsio NIC is better for the same price.