PfSense 10GbE LAN

[Scepterus]

Hey and sorry for the long absence, I’ve been dealing with (unrelated, but actually solved by pfSense) internet issues. Somehow, my D-link router was messing up dialing through the bridge modem router and caused drops in the connection every 30 seconds or so.

It was a long journey and 3 separate technicians visits from my ISP and infrastructure provider (they are separate here, not sure if this is how it works where you live). Until I got tired and just pushed to move the dialer to the pfSense box.

But I digress, I am now fully deployed with pfSense and even found a few good configs for caching common stuff in squid on the pfSense.

I’ve looked at the link you shared, and it speaks only on setting up inline IPS between ports, I do not want that, I want both 10GbE ports to be in the LAN segment and have free traffic between them. Did I miss something in that link that was relevant?

The FreeBSD wiki link is interesting, but it states that I need an 8 core CPU before tuning to get 10Gb.

As for getting SFP cards, do they cost less? And if so, did you mean just use them as is and get SFP cables? Or with converters to Ethernet? I am willing to save as much as I can, since I’ll have to also buy cards for both endpoints, so replacing the cable runs with SFP is not really an option.

On a side note, in case anyone else follows in my footsteps in the future, I did find that there’s a DrayTek NIC that is also a vdsl2 modem that you can use in pfSense. If I was able to purchase that, it would also eliminate my modem and put that into the pfSense box. If anyone knows where I can get that, or something similar, I’d be very grateful.

It’s called the VigorNIC 132, in case anyone is interested.

[MadMatt]

@Scepterus

I’ve looked at the link you shared, and it speaks only on setting up inline IPS between ports, I do not want that, I want both 10GbE ports to be in the LAN segment and have free traffic between them

You want to set up multiple interfaces on the PfSense appliance to be bridged … that’s what 80% of the linked doc is about (with tuning) … you don’t want the IPS part as it will slow things down but you want all the rest …

@Scepterus

The FreeBSD wiki link is interesting, but it states that I need an 8 core CPU before tuning to get 10Gb.

Yes, that’s the established standard, it applies only if you cant’t use dedicated switching ASICs (like the Mikrotiks do) or if, like you stated, you are not willing to push the boundaries and see what’s possible with current OSes/hardware/technologies. I think it will be the basis anyway for whatever experimentation you are going to undertake to get more perfroamce out of your setup

@Scepterus

As for getting SFP cards, do they cost less? And if so, did you mean just use them as is and get SFP cables?

SFP cards cost less, if you can connect them with TwinAx cables, if you are limited to catx ethernet you will need transceivers and it will not be economically convenient and power efficient, hence why I asked if you already had the cabling in place or could make do with additional cabling
SFP+Twinax is the most cost effective/power efficient setup, but you are limited to 5 meters of cables (that cost a fortune, while 1M ones are around 20USD) and you can’t really bend them/run in conduits

@Scepterus

If anyone knows where I can get that, or something similar, I’d be very grateful.

Don’t know where you live, here we get summer (and sometimes winter) thunderstorms … the modem is pretty much what gets fried most of the times so I wouldn’t want to put that in my appliance, even if I could …

[Scepterus]

@MadMatt

You want to set up multiple interfaces on the PfSense appliance to be bridged

Why do I need them bridged? Will putting them just in the LAN group not achieve the same thing? What if I get a 2 port card? Can it do the switching on board and not involve the CPU at all?

@MadMatt

you are not willing to push the boundaries and see what’s possible with current OSes/hardware/technologies.

I am willing to push my current box to its limits. But if it is not capable of 10GbE traffic, I guess I’ll just have to replace the 1Gb switch that’s connected to the pfSense box right now with one that has at least 2 10GbE ports. I saw one from QNAP, but good luck finding anything they make in stock.

@MadMatt

why I asked if you already had the cabling in place

Well in place is a way of putting it, they’re not in the walls or nailed to anything, I just have the CAT7 cables running next to the wall and behind stuff. But yeah, replacing them with optical cables is not ideal. This is also not a permanent setup, so the more fluid the design, the better (in case I need to up and move it to a more permanent place.)

@MadMatt

thunderstorms

Well, can’t say that I’ve experienced something like that in almost 20 years of running on DSL, we had thunderstorms and very close lightning hits, but never hurt the phone lines. They are shielded with rubber on the outside on the poles, that might be the reason.

So bottom line, you would not recommend I go the NIC in the pfSense box route, and instead upgrade my current switch?

[Scepterus]

UPDATE: was just looking at some stuff I saved, and found this:

HP 10GbE 2-port 533FLR-T 701534-001/700757-001 700759-B21 RJ-45 PCIe NIC...

So I ordered one, it’s cheap enough that even if it doesn’t work in the pfSense box, it could in one of the end points.

[MadMatt]

@Scepterus

So bottom line, you would not recommend I go the NIC in the pfSense box route, and instead upgrade my current switch?

If you want the easy way out, and have it working in 10 minutes and have wire speed performance, yes, a CRS305-1G-4S + IN and two 10Gbaset
Transceivers will give you the lowest price possible, if you need more than 4 ports then depending on whether you need VLANs or unmanaged is ok you may get by with an unmanaged QNAP or TP-link or netgear but they will be double the price (and double the number of ports, and no transceivers needed ) …

@Scepterus

Why do I need them bridged?

Because that’s what you asked? using two network cards in pfsense and have it not routed ?

@Scepterus

Will putting them just in the LAN group not achieve the same thing?

If you are talking Pfsense Groups no, they are interface groupings to be used in Firewall and NAT rules, they do not create a bridge, you want a bridged interface to have traffic flow across ywo networ ports without a route:
https://docs.netgate.com/pfsense/en/latest/bridges/create.html

@Scepterus

Can it do the switching on board and not involve the CPU at all?

No, it does not work that way …

[Scepterus]

@MadMatt

whether you need VLANs or unmanaged is ok

Yeah, I don’t plan on using VLANs, and there’s no reason to trunk ports if it’s already at 10GbE. Plus, I hate mikrotik.

@MadMatt

No, it does not work that way …

Well, thanks for clearing all that up. I might just look into an unmanaged switch. Thanks.

[MadMatt]

@Scepterus

Yeah, I don’t plan on using VLANs, and there’s no reason to trunk ports if it’s already at 10GbE. Plus, I hate mikrotik.

Then, if you can find it, your best bet would be a QSW-2104-2T, but I fear that is the one you can’t find in stock locally …

[Scepterus]

@MadMatt

QSW-2104-2T

Actually, I found this one: https://www.amazon.com/QNAP-QSW-308S-Switch-Gigabit-Unmanaged/dp/B07VC9T3WQ/ref=sr_1_5?keywords=10gb%2Bswitch&qid=1643897304&s=pc&sr=1-5&th=1

Plus 2 https://www.aliexpress.com/item/32931277070.html?spm=a2g0o.productlist.0.0.398c4a6a3RcJyx&algo_pvid=f4fe15a3-bd12-4286-9a76-28c1961b6f5a&algo_exp_id=f4fe15a3-bd12-4286-9a76-28c1961b6f5a-0&pdp_ext_f=%7B%22sku_id%22%3A%2210000000410055074%22%7D&pdp_pi=-1%3B137.99%3B-1%3B-1%40salePrice%3BILS%3Bsearch-mainSearch

of these, and it’s still cheaper than that model.

[risk]

SFP+ cards, DAC cables, and switches for short distances are cheaper than RJ45.

You can put a 10Gbps or multigig RJ45 transciever into the RJ45 slot, but the distance will be limited to about 30m (because of power limits on SFP+ slot itself).

Buying preterminated fiber online + proper laser transceivers is also an option for 10Gbps+ or 40Gbps+ for longer distances is not very expensive - just maybe not as practical as using RJ45 you already have

---

Here’s the conundrum that makes recommending 10Gbps SFP+ hard.
There exist 40Gbps QSFP+ nics that aren’t dramatically more expensive than 10Gbps. they’re great for point to point links between e.g. raid array NAS-es.

You could even get e.g. switches with 1,2,4 40GBps QSFP+ ports, and a bunch of 10Gbps or 1Gbps ports (for example: CRS354-48G-4S+2Q+RM - yes mikrotik, but just a switch)

---

If you want to stick to RJ45, a good “long term viable” card is the intel X550-T2; it’s not super cheap, but it’s a PCIe 3.0 x4 card (many older cards like mellanox cx-3/cx-4 you can get for cheap on ebay need 8 lanes because they’re PCIe 2.0 designs); it supports SR-IOV for easy high performance virtualization, and it supports multigig (2.5G/5G).

MSRP is around $350, but you can get them new for $150-$200 at most retailers.

@Scepterus

Can it do the switching on board and not involve the CPU at all?

Drivers for offloading all kinds of functionality on intel chips are generally well supported on linux and freebsd(pfsense).

I don’t know OTOH what kinds of offloading/forwarding is supported on which card. e.g. I don’t know if linux (e.g. with dsa somehow) or pfsense drivers can program the X550 to work as a switch and not bother transfering packets to your CPU at all. Theoretically the chip supports that because it’s needed for virtualization.
eBPF/XDP is also an option theoretically, not sure.

With CPU, you can maybe bypass the firewall between two interfaces, and you can maybe setup irq affinities to ensure you have more cpu cycles across more cores, or fewer cores, thus better use of caches.

Maybe you can increase the PCIe transaction size, to reduce the strain on the memory subsystem.

Maybe eBPF/XDP is something you could do, it’s not an out of the box solution (flow offloading).

Additionally, if you go down the X550-T2 route in the end. You can run a network bridge on linux (more likely to perform better… and if 10Gbps doesn’t work OOB you can try various XDP thingies), and you can carve out an SR-IOV device and give it to a pfSense VM which would give you a nice UI for router configuration.

I’m really curious about that hp/broadcom nic. It’s very cheap, let us know how it works out.

[Scepterus]

@risk

but it’s a PCIe 3.0 x4 card

Well, my board as stated above has PCIE 2.0 x4. So from what you’re saying I’ll need an x8 slot to get a good card, even though the math suggests it is possible to get 2 port 10Gbps on PCIE 2.0.

I just checked, and that HP is also x8, so it will have to go into one of the end points.

I guess if it works well enough, I’ll order another one, and maybe just upgrade my switch when I find something that’s worth it. The QNAP one is fine, but if I order these hp ones, I can get both endpoints connected directly and to the switch in the future.

As for SFP+ not really viable in my case, plus I don’t see my self saturating the 10Gbps connection, even when I upgrade my NAS to TrueNAS with ZFS RAID.

[MadMatt]

For closure, here’s a current article of a guy that did exactly what youe were originally planning - a mixed bridging/routing scenario where he had a quad-port 10Gbit card, and a dual port 25Gbit card (for uplink … lucky ba**ard )

25 Gigabit Linux internet router PC build

TLDR:
with a 6 core 5600x and current generation hardware (and 800USD of network cards) he topped up at 62Mpps e.g enough to support a bridge over 4x10Gbit links
He estimated he would need a 12 core cpu to be able to support 6x10Gbit …

The total cost of the hardware was well over 2000USD …

[Scepterus]

That feeling when his router has better hardware than my gaming PC… well it was a nice enough idea, but it did save me from internet troubles and I learned a lot. Thanks.