PfSense 10GbE LAN

Networking Networking Hardware
1

Hello everyone,
So YouTube has served me a video from 4 years ago where Wendell shows how to turn a small form factor dell pc into a PfSense router.

Now this was almost 5 years ago, so he’s talking about 1 gig speeds and says it would be more than enough to reach it. Obviously, this got me thinking on the possibility of doing the same thing with my setup. Except I was looking at the QNAP QHORA 300 as my next router because it had 2 10GbE ports and that’s basically all I need and I’ll explain.

I have a NAS that I also stream media from, and my gaming computer aside from other devices that are connected either via Wi-Fi or via cable to the router. The main ones are my gaming computer and the NAS, so I want them to be connected both to each other and to the rest of the network with as much bandwidth as possible.

Which leads me to my question. Let’s say I want to create a PfSense box and use it as my router, what’s the minimum specs I require in order to get 2 10GbE ports supported fully aside from the normal operations of the network.

Let’s say I have a first gen i7 hp computer in a small form factor, is that enough? Or do a need a certain number of PCIE lanes to achieve my goal?

I’ve looked everywhere, including this forum, and found no concrete numbers, just estimations and personal anecdotes.

If anyone is interested, here’s the original video:

Thanks and have a great day!

2

It sounds like you’d maybe a switch with a couple of 10G ports and a few 1Gbps ones?

I don’t see why you’d want to bother routing at 10Gbps anywhere in your network.

Your NAS and gaming machine can also be connected directly via 10Gbps while keeping rest of your network ar 1Gbps, with just two NICs point to point.

1 Like
3

I ha

Some of us are gifted with WAN connection >1Gbit.

Other than that I don’t see the point in 10G routing either.

My PfSense VM has 4 Ryzen5900x cores assigned and I’ve never seen load >3% on my 250Mbit WAN. And I also got 4 VLAN’S, opt1+2 going with a shitton of firewall rules and other services. Basic routing is very light on performance. Only if you do VPN encryption on lots of concurrent connections or use other more demanding packages, the load meter might be higher.

Desktop CPUs are totally overkill for this. That’s why any old PC will do fine and why Wendell took the crappiest PC he could find.

4

If you had multiple VLANs and divided them from each other via Firewall, your FW would see a lot of traffic. Not a usecase you would have at home though.

1 Like
5

Yeah, I forgot to mention, my WAN does not exceed 100Mbps, so that’s why I’m talking about LAN only.

It’s not the point, these are basically 2 birds I want to kill with one stone. I want those stations to be connected to the rest of the network with 10GbE available to them, AND I want all the advantages of PfSense as a router (Blocking attacks, managing traffic and so on). Plus, it would give me a great skill and useful knowledge for my work in IT.

I mean, it would cost me more to replace my router and switch with a level 2 managed switch and router, than it would cost me to deck this computer out with network cards.

Speaking of which, I had a question about that. Are there any PCI cards that TRANSMIT Wi-Fi rather than just receive it? If so, that would completely replace my current router.

Are you sure any CPU would be able to handle 10Gb traffic between 2 ports? I mean, I just looked it up, and first gen i7-i3 have PCI-E gen 2 which is 8GBs. So the theoretical limit is 64Gbs per slot, I’m guessing, correct me if I’m wrong.

6

Most cards can do this physically, the firmwares that drivers load onto cards are not opensource and need to be signed, so depending on product segmentation a particular card may not support it… (they want more money to enable the functionality). Beyond that, you’d just run hostapd to manage authentication and VLANs.

Compex.com.sg makes cards that are popular with OpenWRT people who tinker with wifi on x86; there’s others too. Check the OpenWRT forums, or if you’re on IRC, ask in their support channel.

However, most people who tinker with wifi go the opposite direction and run their software on accesspoints instead because of placement flexibility (signal strength) and price. And also because of power use in some cases. For example the unifi u6-lite and u6-lr ($100/$160) can be easily flashed with OpenWRT, similarly you may be able to find Belkin rt3200 for about a $100 on sale, which costs about as much as a compex wle3000h5 - an example card. The u6-lr or the Belkin use arm a53 cores, similar to a pi3.

Firewalling at 10G works the same as firewalling at 1G - there’s no learning there.

A dual port 10G or a dual port 40G nic isn’t that expensive these days, but if you want a bunch of 1G devices to take advantage of the 10G speeds, you’ll need a switch.

On the other hand, if you just care for your PC-NAS, yeah you can get a dual port nic, throw it in between, and firewall between them at leisure.

You can get a CSS610 (or one of the other clones with different software under a different brand name) as a switch off you need VLANs and a couple of 10G ports.

It’s more about the cards, network chipsets, how they handle interrupts across cores, what features the drivers can take advantage of… than raw PCIe lane throughput.

PCIe 2.0 gets you 5Gbps per lane bidi. You’d need 2 lanes (-ish worth of bandwidth). Most cheap cards people would use (mellanox cx-3 off of eBay for example) are x8.

Intel x520 cards are also popular these days.

Hopefully this gives you useful things to consider.
Enjoy,
-s

1 Like
7

Thanks for the reply, regarding the Wi-Fi, it’s good to know it’s possible. The reason I was thinking of this, is because a PC has more power and cooling to drive the antenna better than what I have now, which is a d-link router with ac Wi-Fi.

I’ve looked into the card manufacturer you mentioned, but they only use the m.2 slot from what I saw in the pictures, and not a PCI slot.

Regarding the firewall, again, 2 separate subjects. I want to learn to optimize a PfSense box and get it running, while also not getting a 10GbE switch, which are impossible to find right now.
My objective is having that box be a centralized hub for the networking, so I could get rid of some of the switches in my config, while also having the option to upgrade anything later on.

I guess I forgot that these network cards actually handle some of the computations on board, so does this mean that I can get a dual 10GbE port PCIE card and get the full bandwidth out of it? EDIT: scratch that, just noticed you said it was per lane, so duh.
Also, if it is, do you recommend any low profile ones that would fit in a small factor case?

I also want to keep it strictly Ethernet and not SFP, as I already have CAT7 cables connecting my existing network and this would save a bit on the expense.

You have been very helpful thank you.

8

Antenna power / signal strength is limited by legal regulation and certifications, you should generally expect the “PC as access point” solutions to be slower.

This is why I mentioned flexibility of placement; it’s free decibels. (The u6-lr are quite good btw).

2 Likes
9

So I got the computer, it’s a small form factor hp 8100. pfSense works on it, I’ve put 16 GB of ram in it, just because it’s 1333 ddr3 and I have nothing else to do with those 4gb sticks.

Now regarding the 10GbE expansion, I noticed I have 2 full length PCIE 4x slots, and 1 PCIE 1x slot. And another PCI slot that I already put a 1Gig card, so I could have WAN and LAN interfaces.

Given that this is a small form factor case, which cards can I actually fit in it and hopefully get 2 ports per slot, just in case I’d want to upgrade in the future?

Also, I’ve already installed pfBlockerNG and snort to test them out, I already have a pihole as my DNS and DHCP, so they are just for attacks coming from the outside.

Is there another must package you would recommend?

10

I run a home server as a NAS / Plex / etc. box with dual 10GbE using an Intel X550 NIC. The server is on a 10GbE switch along with my office, where my workstation also runs dual 10GbE. This is all on the same LAN, so no routing necessary. I have a separate PFSense router running on a Celeron J1900 with 1Gb WAN.

Netgate developers seem pretty skeptical that it’s even possible to route at 10Gb with pfsense, hence why they developed TNSR on Linux/DPDK, but you ought to be able to get close with the right hardware. There’s a thread on reddit discussing this, search for “hardware_validation_for_10_gbe_wan_pfsense_box”.

1 Like
11

Again, I don’t plan to route 10GbE out of my network. It’s just a cheaper alternative to upgrading the switch and getting cards for the endpoints.

12

If your goal is indeed to improve knowledge of your work in IT, then the takeaway is that you shouldn’t even be thinking of routing 10GBps over multiple interfaces without considering a dedicated appliance.
Unfortunately for you, unlike 10Gbps switching that is served very well by the Mikrotik/TP-link/Ubiquity low cost gear, 10Gbps routing still requires datacenter hardware to achieve line-rate, or stupidly expensive CPU power that couple dwith power requirements makes the dedicated solution economically comparable.
So, in short, get the cheapest 10GB switch you can find that suits your network cards if you already have them buy cheap connectx SFP+ ones with a mikrotik switch if you don’t have them and use your NAS as it is supposed to be used.

If you want to ‘experiment’ then it is all well and good, just do not expect a lot of the ‘experts’ to chime in, as the setup you want tro try out doesn’t make sense either from an architectural or economic point ov view …

13

Well so far all the “experts” here seem to miss the point of this thing entirely and just come to post on telling me what I’m doing wrong and why they think it’s pointless.

I do not agree that routing should be left to appliances, from my experience with routers in general and mikrotik specifically, they are terrible at their job from a software perspective.

I’m not trying to achieve 10GbE routing, just switching between interfaces. That can be done mostly on the NIC’s.

If anyone else feels the need to come here and post about how insanely pointless this thing or other is, please refrain from doing so. Also remember that @wendell is always doing stuff that’s insanely over the top and pointless to most people, but that’s the best way to learn for some people.

14

Sorry, maybe I came off too harsh from the start … monday morning, kids in home schooling because of COVID … not enough coffee…
I am by no means trying to say that what you want to try is pointless, at least to me it is still not clear what you want to try because in the various messages there’s confusion between routing and switching …

Switching is generally left to appliances, because they can nowadays achieve line-rate between ports with a very few components and a very low Wattage, unless we’re tallking 10GBaseT but the wattage there is because of the media format, not because of workload on the SOC/ASIC that does the switching.

I am not overly familiar with routing enterprise gear, but I am assuming most all of them will run some sort of Intel/ARM based CPU, maybe assisted by specific asiscs for offloading things like encryprion and such … the amount of CPU power/watts needed to support 10Gbps routing, especially when you add packet filtering and maybe some security checks on top becomes pretty quickly ‘interesting’

Mikrotik routers are a weird concept whereas unlike the other ‘enterprise’ vendors that unlock routing platform features in progressively more expensive hardware they provide all the routing features for all the hardware they sell, and they run specific training courses in what each hardware platform is suited and unsuited for, but people just look at the low price, tries stuff the docs warn you not to do , and then complain because perfromance suck …

Anyway, you want to leverage some of the NICS on your NAS to avoid buying a dedicated switch, and that’s easy enough, just dedicate some of the ports on the NAS to that function, bridge them together, and connect stuff … provided you don’t mess with iptables you will get a decent performance … how decent will depend from the nic brand and how well its driver is able to process layer2 frames without bothering too much the cpu.

But then, later on, you say you want to deploy PfSense … and that’s where I at least got confused and assumed you were in fact talking about routing.

You could in theory deploy a pfsense VM on your NAS, pass through some NICS, and then use PfSense gui to define separate or bridge LANs … that is the case where everyone is warning you not to expect too much perfromance/you would be better off with a dedicated and different solution … (and this from someone that has been running exactly that - Pfsense in a VM hosted on TrueNas - for years now)

So if you can expand a little more (maybe with a diagram) on what you are trying to achieve, we may be better suited to try and help you along instead of telling you that something can’t be done …

1 Like
15

My own personal experience is that you need a high clock cpu to handle 10gbps.

10gbps switching isn’t suuuper bad. What has overhead is routing and packet analysis to the tube of 10 billion bits per second.

Your processor only cycles at 4-5 billion cycles/sec. That does add a fair bit of relative processing latency and overhead

Newer nics that offload more function to hardware can work better. It treats the data as flows instead f individual packets.

Ymmv etc

2 Likes
16

Hey it’s fine, no harm done, it’s just that a few other people also went that direction, I guess my wording could be the cause.

So I’ll start from the top, when I posted this topic it was a theoretical idea that I wanted to see if it was possible, but now I actually have the system deployed and running tests.

The main goal right now is this: VDSL to pfSense box WAN port (1gb) pfSense box to normal 1Gb switch for the rest of the LAN and Wi-Fi, 2 10GbE ports on the pfSense box running cat7 to 2 machines, 1 NAS/Server 1 Main station.

The pfSense box has the following specs:
HP 8100 SFF, i5 650 3.2GHz, 16GB DDR3 1333MHz and right now 1 external PCI (not express) network card 1Gb.
The PCIE lanes are 2 4x full length slots and 1 1x slot.

I’ve previously deployed a pfSense box for testing in my workplace, that was to replace the fortigate we had that was ISP managed until we had a breach and I got the access to lock it down.

But then COVID happened, and I left that job, so I didn’t get a chance to fully explore the applications of this software. In my last job, I inherited a mess of MikroTiks that were poorly configured, and I could not make heads or tails of the documentation and the interfaces.

Consequently, I was blamed for a mess with the network I solved, and that’s one of the reasons I don’t work there anymore. So yeah, I’ve had bad experience with these appliances.

I’m not completely without training, at the start of my career I went through a few certifications such as CCNA, CCNA security and JNCIS, so I know my way around their software and logic.

Hey Wendell, thanks for chiming in, would you say it’s feasible with the specs I’ve listed to get a decent NIC to handle that, or should I settle for 2.5 or 5 gb?

I’ve looked into a 10GbE switch and router from QNAP, but they are virtually out of stock everywhere, so I figured instead of letting this hardware go to waste, maybe I can reuse it in my network.

Thanks, and I hope this post clarifies most of the things.

17

Do you already have the 10Gbe card/cards? If so, what model do you have?

18

I do not have ones, I was hoping for a recommendation for ones that would work and also fit the case, as it is low profile. What you quoted was the current goal, I’m 75% there, just need the 10GbE cards on the pfSense box and the endpoints.

19

You want to document yourself on how bridging can be set up on freebsd/pfsense, some hints could be found here:

in general, high performance settings for nics in freebsd:
https://wiki.freebsd.org/Networking/10GbE/Router

As for what will work in your setup … you will need to test and see if it works … make sure you get something intel, as the cheaper broadcoms may not support being configured with netmap in freebsd. Given that you need to buy, are you sure you don’t want to use SFP+ cards with direct attach cables?

1 Like
20

Another interesting read:

1 Like