After watching the Son of the forbidden router video this topic intrigued me. The build Wendell went with is probably fine as is, but the cost of the motherboard he used seems quite unreasonable where I live (around 500€).
I currently have a Protectli type SFF-PC running OpnSense. My plan is to run OpnSense on bare metal in the future too. My goals are quite simple: 10Gbe or close-to routing and manageable power consumption overall. Idle consumption is a big part of this picture.
What I have planned is something in the lines of:
i3 12100
Some generic B660 motherboard
Some small good power supply
2x8Gb DDR4
NIC or two
Am I on the right track? I’m willing to forgo ECC in my router, but if there is a comparable solution with ECC that is ofcourse better alternative.
FAQ: Why are you not virtualizing the router?
I really do prefer a separate router for my homelab. I am aware of the ability to virtualize the router too, but I prefer not having everything tied to one physical machine.
Can’t you reorganize your devices and VLANs so that you won’t need 10Gbe routing?
Truth be told, I probably could and it’s not like the 1Gbe bottlenecks are killing the performance now. This just something that I have wanted to try for the flexibility it gives. No matter what I’ve done I’ve ran into this specific cross-VLAN traffic routing hindrance is some capacity every time.
My pfsense VM with 4 cores of i5 12500 and vmxnet3 NICs managed just under 9Gb/s of iperf3 (10 streams) traffic across VLANs. Hypervisor is running 2 sticks of 16GB DDR4-3600 16-18-18
On bare metal you can probably route pretty close to 10Gb/s assuming your packet sizes are close enough to 1500 bytes.
It “idles” at ~50W (measured using a kill-a-watt) due to the CX3 I have in my system. That CX3 prevents the CPU from going into C states deeper than C2. Idle in my case meant ~5 VMs running on esxi doing mostly nothing.
I’ve seen people report 30W idle measured at the wall for Alder Lake.
The 30W idle was something I had found earlier, but your experiences with the Mellanox card was new to me. That is quite a bump in idle usage with just one network card. Do you have any idea if other cards exhibit the same behaviour?
After watching the video linked in the OP, I am wondering if the new 5600X3D chip just released, with some ECC RAM would be a good platform for this application. I’d think the huge cache could be a big benefit.