10Gbe router build for home use, recommended hardware?

After watching the Son of the forbidden router video this topic intrigued me. The build Wendell went with is probably fine as is, but the cost of the motherboard he used seems quite unreasonable where I live (around 500€).

I currently have a Protectli type SFF-PC running OpnSense. My plan is to run OpnSense on bare metal in the future too. My goals are quite simple: 10Gbe or close-to routing and manageable power consumption overall. Idle consumption is a big part of this picture.

What I have planned is something in the lines of:

  • i3 12100
  • Some generic B660 motherboard
  • Some small good power supply
  • 2x8Gb DDR4
  • NIC or two

Am I on the right track? I’m willing to forgo ECC in my router, but if there is a comparable solution with ECC that is ofcourse better alternative.

FAQ:
Why are you not virtualizing the router?
I really do prefer a separate router for my homelab. I am aware of the ability to virtualize the router too, but I prefer not having everything tied to one physical machine.

Can’t you reorganize your devices and VLANs so that you won’t need 10Gbe routing?
Truth be told, I probably could and it’s not like the 1Gbe bottlenecks are killing the performance now. This just something that I have wanted to try for the flexibility it gives. No matter what I’ve done I’ve ran into this specific cross-VLAN traffic routing hindrance is some capacity every time.

1 Like

My pfsense VM with 4 cores of i5 12500 and vmxnet3 NICs managed just under 9Gb/s of iperf3 (10 streams) traffic across VLANs. Hypervisor is running 2 sticks of 16GB DDR4-3600 16-18-18

On bare metal you can probably route pretty close to 10Gb/s assuming your packet sizes are close enough to 1500 bytes.

1 Like

Sounds like I was quite near with my initial guess.

What is your systems power consumption? Have you measured the consumption with just OS disk running?

It “idles” at ~50W (measured using a kill-a-watt) due to the CX3 I have in my system. That CX3 prevents the CPU from going into C states deeper than C2. Idle in my case meant ~5 VMs running on esxi doing mostly nothing.

I’ve seen people report 30W idle measured at the wall for Alder Lake.

The 30W idle was something I had found earlier, but your experiences with the Mellanox card was new to me. That is quite a bump in idle usage with just one network card. Do you have any idea if other cards exhibit the same behaviour?

Find something that properly support PCIe ASPM. Afaik Intel cards do. Mellanox not so much.

After watching the video linked in the OP, I am wondering if the new 5600X3D chip just released, with some ECC RAM would be a good platform for this application. I’d think the huge cache could be a big benefit.