Hi everyone! Long time lurker on social media, barely active on Reddit, looking for some advice into building a long lasting networking solution.
I have been looking to build a home router that is 10Gbps capable, as I want to futureproof my network setup (i.e. build it once and don’t touch it in 10 years, for the most part). While I already have looked at options for Access Points and Switches (mostly Ubiquiti), I am still unable to identify a good solution for a home router. Here are my considerations below:
Build it myself, as I will have to have more control over the OS that it runs
As low-powered as possible, because electricity is going up by the day
Connectivity: Want to have a mix between SPF+ and 10GbpE RJ45 links, kinda like the Netgate 6100
I was originally thinking on getting something like a Thinkcentre (from the STH Tiny Mini Micro project list) but my concern is that I haven’t seen anything that allows for appropriate PCIe expansion for SPF+ connections… or maybe I haven’t understood the specs appropriately.
Any suggestions on what type of reliable hardware I can get for this sort of purpose?
How much are you paying per kWh? I used to pay $0.34 per kWh and I moved so now I pay $0.10 kWh. I remember the pain of living with expensive electricity.
I do agree that power will keep going up. I’m leaning towards repurposing a Dell r230 to a pfsense server. With 2 SSD’s it averages 30 watts.
As for your choice, I’d be looking at equipment that has a PCI-E slot so you can add your network card. You’ll need to make sure your card and OS play nicely too.
If you’re into the homelab and want to expand it might be worth looking at solar options to help offset the cost too.
Depending on how much processing power you need for routing and firewall rules Dell R210ii are usually pretty cheap and have a PCI-E slot for a 10G NIC if you want rackmount.
If power efficiency is a serious concern then building something with a current gen i3 is a good option or buying a netgate or aliexpress appliance that is ARM or low-powered x86 based. I believe Serve the Home does a lot of content around these devices.
Check the ThinkStation/ThinkCentre megathread over on STH. There are several builds (mine included) using dual SFP+ NICs in those machines to build space-and-power-efficient firewalls or hypervisors. My firewall runs on a pfSense virtual machine within Proxmox, and uses about 17-20w during normal usage running only the pfSense VM. Adding a couple other VMs and the power usage goes up a few watts. My recollection from installing pfSense bare metal on these devices is that they’d idle in the 12-14w range, but I can’t remember if that was truly idle or if they were working as a firewall/router at that time.
Absolutely! My firewall host is a Lenovo ThinkCentre M720Q with an i5-8500T, 32GB DDR4, a Supermicro AOC-STGN-i2s 2x SFP+ NIC, a 256gb SATA SSD boot drive, and a 512GB NVME. Running Proxmox 9 and currently idling between 20-23W running a pfSence VM, a Plex LXC and a Debian VM running grafana and some random dbs. Actually need to convert that one into docker containers at some point. With the NICs virtualized it will not route 10GbE at linespeed, more like 4-5Gb/s. For the VLANs between which I care about having linespeed routing, I let my L3 switch handle the routing. Passing the NICs through gives another 1-2 Gb/s routing performance, but I honestly don’t care about linespeed routing for the VLANs that need to touch the firewall to speak to each other. And by using virtualized NICs the firewall VM can quasi-failover to another Proxmox node when I need to do maintenance on the firewall host. Overall, I love it and I’m really impressed with Lenovo’s hardware engineering on these little boxes.
Amazing! - Right now I’m looking for a pure router (barebones) as I run everything else off a dedicated server. Now I see that the VGA blank plate is where the PCIe cards go, and I’m assuming it takes half-height, and I should be looking at one of these TMM machines + the PCIe adapter off reputable stores (hehe) + a 2 SFP+ NIC … I don’t have that much need for an intensive firewall (yet).
One last question - did you have to put a heatsink for the NIC? I’m just concerned about the heat in these tiny machines
Yes, the PCIE card bracket sits on the top right as you look at the rear of the machine. It must be half-height and it must be pretty short. That’s why I love the AOC-STGN-i2s (v2)–it uses a tried-and-true chipset (Intel 82599) and it’s tiny. You need the correct PCIE riser for your specific generation of Tiny. The riser for the M720Q/M920Q use a different riser than the M90q/P340 for example. You will also need the rear bracket (Lenovo for some weird reason calls them a baffle). Check the initial post in the STH Lenovo Tiny megathread for info on the part numbers, etc.
Let me know if you have any other questions–I’m happy to help!
EDIT: Oh right heat. Unfortunately, the one shortcoming of the Supermicro NIC I’m using is that it does not have a temperature sensor, and it does get toasty inside the little case. Thus far I’ve been YOLO-ing it and it’s been fine for more than a year, but figuring out a tiny bit of airflow for the NIC would not be a bad idea.
Yeah I just found out about the Baffle situation reading the STH thread - makes sense that Lenovo does this, [sarcasm] but does it really?? [/sarcasm] As for the heat, I will have to check on some ventilation shenanigans somehow. In any case, I will look for the right parts to buy, and will update on results!
This is part of my own pet project on getting a 10Gbpe ready environment - futureproofing myself for the foreseeable future. I already have a Unifi AP (Wifi 6E - should last a while, backordered a 10Gbpe switch (SPF+/Eth) from Unifi as well, so just missing the router component to be done with the entire solution (and time, that is).
And a side note: I am running a Unifi controller from a Hyper-V VM in a separate box, but been wondering if it’s truly necessary, as I can try to build a “forbidden router” of sorts with this mini PC, but is it a good idea? I will always have the Hyper-V server handy… I shall look into it later down the road!
IMO running other network utils on your router box is a pretty solid use-case for the “forbidden router.” In the past I have run my DNS/PiHole and unifi controller in a debian VM on this very same host and it’s worked great.
Depending on your precise needs for switching, I’d recommend checking out the (truly gargantuan) Brocade switch thread over on STH. I’ve been running Brocade as my core switches for 3 years now (ICX6450 and now ICX7250) and they have been absolutely incredible. L3 routing at line speed plus all the POE you want. The only potential downside is that they are not silent unless you replace the fans. They are not loud (except at initial power on), but they are audible if you’re nearby. Oh, and I suppose they’re a bit thirstier on power than I’d like (the 7250 more than the 6450 in this regard).
Wish I could run the Brocade Switch, but that’s truly truly gargantuan (outrageous!). I virtualize all of my servers and similar stuff, and right now I’m a bit intrigued by the loss of speed on virtualization. Perhaps it would be useful for me to pass on the entire SFP+ card to the VM. I will have to ponder about this
The 24 port ICX6450 is pretty much the same size as the smallest rack-mounted Unifi switches. Sounds like your switching is not rack-sized, however…
You’ll get closer to bare metal performance by passing the NIC through to your firewall VM, but you lose some of the benefits of virtualization (primarily migration/failover). On the other hand, if you just want a single box to run your router but also a couple of small VMs, then passing the NIC through might be a good choice. Do be aware that if you pass the NIC through you (obviously) can’t use those interfaces for your other VM(s), so you’ll be running your VMs over the built-in 1Gbe NIC. Not a huge deal; just something to note.
Yeah, my networking stack fits in a “IKEA Kallax shelf” worth of space! Don’t really have the need to do that much networking cabling, as all of the clients I have are wireless, only my main server runs directly off the dedicated switch right now
So I decided to bite the bullet and start this project going - I got a M90q Gen 3 (12th Gen) as I am interested in this thing being as power efficient as possible, and the E-Cores can certainly help at that. Got the appropriate PCIe Riser and bracket, and going also with the AOC-STGN-i2s. Once I get everything to assemble the unit, I will get the SFP Transceivers and that should be it! I already upgraded the rest of the network to 10Gbpe so this is the only component missing.
@teltersat Hi, apologies for bringing back an older thread but i recent purchased the M90q gen 3 as well and am doing a similar build to yours. I found a compatiable PCIE riser on lenovo and wanted to check with you on the model you got and if it works with SFP+ cards… model I purchased:
