I have been slowly upgrading my home network to support 10Gbit internally, and possibly support >1Gbps Internet speeds
I am currently running a couple Mikrotiks in various parts of the house, top of rack is a CRS312-4C+8XG in the attic, together with a cisco SG200-26, studio and basement have CSS610-8G-2S+s they provide connectivity to:
- 10-20 wired LAN points
- various IOT appliances
- two Ubiquity APs
I am running two Microserver Gen8s in the attic with truenas, to support various dockerizez apps and a pfsense virtual firewall, both of them are still connected with 1Gb links
I have been looking into options to upgrade at least one of them (the second being a backup) to 10Gbit, mainly to be able to support, eventually, WAN speeds over 1Gbit - fiber is deploying slowly, but it will arrive, and when it does it will be 2.5Gbps to 5Gbps so even if it is a couple of years off I want to be prepared 
My budget … not the 2K needed to upgrade a Gen8 to a truenas Mini XL, but conscious that 10Gbit parts even if affordable, do not come cheap.
I ‘think’ I scored a deal and I bought an old (same generation as my Gen8s) Supermicro X9SRi-F for about 300EUR shipped (thanks Brexit, could have been 230 without duties).
The motherboard came with a XEON E5-1650 V2 6 core/[email protected] processor and features:
- 2x Gigabit Intel NICs
- support 256GB of DDR3 ECCram or 64GB non ECC
- 6 SATA ports (2x 6Gbps, 4x 3Gbps)
- 4 SAS/SATA ports (6Gbps)
- IPMI
- 1x 16 PCI-e
- 1x 8x PCI-e
- 1x 4x PCI-e
It apparently supports bifurcation as well, will try it out with a cheap 4xNVME card I had in the parts bin
Also, from the parts bin:
- 4x8GB 1666 DDR3 non ECC RAM
- 2x 1TB Nvme
- Bequiet 650w Dark Power PSU
- a noctua 120mm fan
- a ‘sleeper’ case from the 90s
So far so good, Truenas installed, boy the Java based IPMI KVM sucks hard , bifurcation appears to work (but I only loaded one NVme in the expansion card), waiting for a 10Gbit card, the only other piece I plan on buying for now (250EUR new, tried a couple sellers on Ebay but between fakes and people not shipping the items I got fed up and bought new)
The plan, when I am done with tests, is to retire and sell one of the Gen8s, will probably be able to make 350EUR or so (It is a Xeon model with 16GB ECC, leaving in a 3TB WD red for data, probably will install ESXi to make it more worthwile).
This, barring surprises, would bring the ‘upgrade’ cost to 200 EUR, or 300 if I decide to splurge on 64GB of ECC memory, we will see…
I am also planning on retiring pfsense and going with VyOs, I have been running some tests and it looks like it will be able to do all the things I currently do with PfSense, and then some, and possibly support 10Gbit routing to my WAN, if needed. I have been using pfsense for almost 12 years now, but I don’t really like the new development cycle, plus freebsd is really behind on almost all aspects I use in my firewall, and would definitely struggle to get 10Gbit Layer 2 performance with this hardware. It looks lie Vyos will also support IPv6 natively on my Starlink WAN … we will see…
It also looks like my power budget will not increase … the Gen8 with 4SSDs and 2x10TB WD reds pulls 60-70Watts at idle, the Supermicro is currently pullling 30-35 (4 SSDs, 1NVme, no HDD, no 10Gbit card) … I would be suprised if I get over 50 … but we will see …
