So my home server is getting seriously bottlenecked/slammed (traffic overflow notices and lagging out) by the internal gigabit NICs. I thought about NIC teaming or bonding but that would require a switch that supported LACP at the minimum. Then the option of 10Gig appeared. The 10G NICs are not that expensive but what I need is a switch that supports 1-2 10Gig links that wont cost me an arm and a leg.
At least 1 10gig port. If I have 2, I may invest in a 10gig NIC for my desktop as well and run direct copper RJ45 10Gig to it as well.
I got a Netgear GS110EMX . Checks all the boxes for a small home network. No fans, plug&play, VLANs, LACP. Just works.
Zyxel has an equivalent but don’t have any experience with Zyxel outside of a 33.6k modem 20 years ago. But 4+2 or 8+2 switches are rather common. Just check on fans and power, lots of old expensive models out there.
10Gbit RJ45 ports are rather expensive. And they run very hot. The ones with >2 ports use fans and get really expensive really quick. I wanted a cost-efficient managed option and wasn’t disappointed.
If you don’t have any 10G hardware yet, you can still check if SFP+ is a better option. I just used my old cables and I didn’t want to deal with transceivers and fiber cables just yet. But SFP+ switches are damn cheap. I’d check Mikrotik if you want to go SFP+ or larger switches.
1 Like
No, it does not. Only mode 4 (below) requires switch support. Mode 6 performs best if you don’t have any problems with it in your environment.
0 balance-rr
1 active-backup
2 balance-xor
3 broadcast
4 802.3ad
5 balance-tlb
6 balance-alb (charp)
10G switch $117: https://www.amazon.com/Mikrotik-CSS610-8G-2S-Switch-Gigabit-Ethernet/dp/B093QJCWBT
10G card: $16: HP StoreFabric CN1200E 10G Dual Port Converged Network Adapter Card E7Y06A | eBay
Requires either twinax SFP cables, or fiber optics.
3 Likes
Mellanox MCX311A-XCAT ConnectX-3 EN 10G Ethernet 10GbE SFP+ PCIe NIC w/2 Bracket | eBay?
Mellanox MCX312B-XCCT CX312B ConnectX-3 EN Pro 10GbE SFP+ Dual-Port PCIe NIC | eBay?
https://www.amazon.com/MikroTik-CRS305-1G-4S-Gigabit-Ethernet-RouterOS
MIKROTIK Cloud Router Switch 1x GLAN 4x SFP+ RouterOS L5 (CRS305-1G-4S+IN) | eBay?
Gets you going for as little as about $200 (before DAC or SFPs) and still has great linux support and I think even windows support.
The MCX312B runs cooler than the MCX311A’s but costs a bit more.
1 Like
so, how would I know if I have problems with #6? what is the difference between 5 and 6? and vs 0 and 2?
I was just going to link that same switch from Mikrotik. I’m using it here since after scaling down the homelab I didn’t need a bunch of 10g ports. I could just do a direct connection between workstation and NAS, but I’d rather configure what I need to at the switch than have it spread out across different machines.
The CRS305 is a nice little switch also if you need more than two ports. Plus, you can get an external power brick for it in case you don’t have a PoE switch or any injectors.
Go with Mikrotik if you want something cheap but still in the realm of affordable.
My experience with Zyxel are their consumer firewalls and routers and these are mostly trash tier.
I use a Unifi Flex XG which is passive cooled and has 4x 10GB ports and 1x POE-in/1GB port. But with 400 Euro it might out of your price range.
problem appears to be resolved but we’ll see.
I replaced all the cables (LAN1 and LAN2 and BMC/IPMI) with CAT6A cables and the lights cleaned up and ethtool reports gigabit connection.
I also setup balance-alb bonding in Cockpit. I am currently testing by transferring several multi-gig files from the server to my nvme. So far, I am seeing 500+ Mbps over the LAN from the server on one NIC and 3 Mbps up from the other.
If it fails again, I will buy that Mikrotik and jump down the 10G rabbit hole. Especially since I would need to decide between DC 10Gig or LC-LC 10Gig along with picking the correct SFP+ modules and 10G AIC for optimal compatibility with the Mikrotik.
I am currently seeing transfer speeds in Filezilla of 58.3 MB/s whereas before I was only seeing a max of 10MB/s.
Don’t stress to much about compatibility I have some junk no brand sfps in my mikrotik switch and they have been fine.
As for cables vs fiber it’s really dependent on distance. I run a mix and a rj45 for 10g over copper.
I’m also using a mix of 311As and 312Bs with the same mix of adapters other than the rj45 since that is connected to my Dell workstation.
A single gigabit link will max out at 120MB/s
If you are doing 58 you are not maxing out one link let alone two …
If you were seeing 10MB/s either you were linking at 100Mbit or what you thought was network congestion was something different
1 Like
i didnt think it was network congestion. I figured a bad cable or a bad NIC. But, I also figured that if the NIC was bad, why not just go 10 gig so I wouldnt have to worry about congestion in the future when I upgrade my ZFS array from HDDs to SSDs. I also will probably replace the CAT5e connection on my desktop with a CAT6A as well. That may assist with transfer speeds.
If you need a router and not just a switch, but do not need a lot of ports, having a computer as a router can be pretty decent as well. Two or four ports as 10G and four more as 2.5g. Then you can get away with a couple of inexpensive NICs in an inexpensive machine, or one built from spare parts.
Might still come out cheaper than most 10G router/switch units.
Not likely.
$85 Cisco 4948e (4x 10G, 48x 1Gbase-T, L2 switching or L3 routing): Cisco WS-C4948-E Switch Dual AC Tested 30 Day Warranty! 746320908885 | eBay
Rather noisy, though.
Power draw will make this way more expensive in the end. Switches run 24/7. Even if I got that Cisco switch for free, I wouldn’t use it.
More expensive as compared to what? Did you see the comment I was replying to?
Certainly less power than a full PC with NICs left, right and center. Both options can’t really compete with small modern gear when it comes to power. Quad SFP+ Mikrotik or other passive consumer stuff with 1-2 10G ports is just more economical sound in the long run.
I would imagine a very modest computer with a dual port 10G and a quad 2.5G could do a decent job if I am properly guessing at your needed port count and performance. Could be quiet and low power if you build for that. A fair few people have a few years older computers lying around that can the job and just need to pick up a couple NICs and setup the OS. Depends how adventurous you feel.
If you get anything closer to a mainstream application or near the cost of a newer gen 10G router, building probably wont make sense. FWIW, just trying to help.
This is not true.
4 10Gbit ports plus 4x2.5Gbit ports means your ‘inexpensive machine’ would need to be able to software process 50Gbit/s worth of L2 traffic to match an equivalent switch.
Doing that in software, as opposed of doing that using a switch ASIC will consume 10x the amount of power and will require a modern Xeon running at 3Ghz or more …
If on top of that you want to do routing then all bets are off on what actual packet per second/bandwidth your ‘inexpensive computer’ will be able to process, it may not even get to saturate a single gigabit link if you go lower than a modern quad core …
Recent switching hardware has started recently to be available at reasonable prices for 2-4 10Gbit links and plenty og 1Gbit ports, models with 2.5Gbps ports have started to appear but my guess is it will take one year still to be flooded with models that can provide 4x2.5Gbe, 2x10Gbe and run fanless and draw less than 15W of power if you don’t have crazy long 10Gbit cat6 runs … we will see …
1 Like
My point being, this is not the moment to be ‘future proofing’ a home network for future 10gbit speeds.
If you need 10Gbt speeds now, you have cheap and low power options for 4x10Gb links (mikrotik, sfp+), or 2x10Gb links and 6-8 1Gbit links (every vendor), 1x10Gb link and 8x 2.5Gbe (recently reviewed on servethehome)
If you need more, you have the Mikrotik CRS312-4C+8XG-RM that, for 500USD gives you 12 10/5/2.5/1gbit ports, 4 of which are combo SFP+, it will draw 30w and is not silent.
There’s something equivalent from qnap, but costs more (900USD) and makes less noise
(QSW-M1208-8C)
Any option with more than 4x10gbit ports is still expensive
If you want to spend less for a similar amount of ports you wil need to buy old enterprise (arista,dell, brocade) but they will tipically have SFP+ and gigabit rj45, you will have to find the right SFP+ transceiver to get 2.5 gbit, it wil get expensive quickly, they are loud as heck, ad draw anywhere from 70-200W of power, but they will have Layer 3 routing/bgp and other enterprise goodies … for as low as 200USD for a brocade 48 port with 16x 10Gbit SFP …
The only combo that so far has not surfaced and every homelabber would die for is a 4 10Gbit SFP+ with 4 additional 10/5/2.5/1Gbit copper and another 8 to 16 2.5/1Gb combo port, possibly with POE, that runs fanless, draws less than 30w and costs less than 300USD …