Just got my Internet connection from my ISP upgraded to 25Gbps (There is no typo there), and now I’m facing rich kids problems. It seems I’m getting some BW limitation, most likely due to hardware. That’s why I’d like some of the communities views on that one.
My hardware/software setup is : Elitedesk G2 i3-6100 + Mellanox ConnectX-4 + Proxmox - Interface bridge to a Debian VM.
I suspect that my limitation is my CPU clock/speed, and I’m considering upgrading to I7-6700 (that’s what the Elitedesk G2 is supposed to sustain), and didn’t tried my Ryzen 5600X yet to see what was the efficiency comparison.
Otho, I know that most of my tests (speedtest/iperf3) were not optimized for that kind of loads (multicore load- Any ideas there ? ). My end target is to NAT on my GW (I’m platform agnostic, but it’s still a " home " setup, so Intel / AMD is an option, noisy fan aren’t.)
I didn’t find yet an option for fanless SFP28 25G/10G switch (I know there’s a Mikrotik out there in the wild that will accept SFP28 fanless., but it’s categorized as " Router " ).
Ho and for reference, I was able to speedtest up to 12Gbps and Iperf3 up 17Gbps (multiple servers) with this CPU.
Any thoughts ?
PS: The uplinks/path toward the speedtest/ipref are multiple 100G and known path, so I should have a decent metric for measurements, I’m open to debate this around PMs.
Switzerland, Init7, 777CHF per year (roughly 840USD on today’s exchange rate). Also note that there I had to pay a fee for the upgrade to 25G, roughly the cost of the optic in the POP. No CGNat, IPv6 prefix delegation, CPE agnostic. And I’m not working for them, they are friends, I work for another ISP also in Switzerland (just so it doesn’t look like advertisement ) .
Thanks for the link, already was aware of Michael’s blog post, I just wanted alternative feedback.
In the interval, I took a chance to drop the Connectx4 in my ryzen, and could directly reach 23 gbps download test with a low cpu load from the gateway without any major tweaks. So that’s a quite good sign that the poor i3 is a bit low power to handle those 25gbps.
I’ll continue my exploration and see what are the best settings for efficient NAT.
Good point, I also thought of that option, as the i7-6700 is supported by the ElitedeskG2. I looked up the used market prices (available around here) and the delta with a more recent AMD Ryzen 5 new is so low that I’ll probably build a new machine and reuse the old thing elsewhere.
I swapped the fans and added some Noctuas, got a Kioxia boot nvme, and a pair of Samsung Evo for system drives for a Proxmox and VMs. I’m still reaching my performances, and have a power consumption that is " only " 50w ish. That’s a perfect compromise for my needs. And I resold my Ryzen 5
How interesting that you are able to hit 25gb with lower clocked cores. I guess the system is managing traffic in a well threaded way to make use of multiple cores. So it was more of the i3-6100 being a dual core that was the problem rather than clock speed. That Epyc is also much slower than the Ryzen per core was, so really it seems like for these high bandwidth routing needs cores always wins over clocks.
So I made even another update on that topic : I tried an I3-13100, and could also speedtest / iperf at roughly 23-24Gbps. I don’t have any specific writeups and results, my time was limited. Power user went even lower than that.
I also tried that R86s that Patrick from STH tested some time ago, I know that it will not be able to nat 25Gbps, as the bus to the NIC is only 4 PCI lane, but my 25G link is at least up
Regarding the clocks speed, I suspect that we could think about VPP and offloading some workload to the NIC directly using DPDK. However I’m not experienced enough on that side to make real world tests / statistics on that (yet)
I hate you with the white hot intensity of a thousand exploding stars. Am paying $3,120 a year for 750mb/75mb, which seldom goes over 500mb. I miss being in a civilized part of the world. Jealous AF.
Init7 and us (my day job) are both doing fixed IPv6 /48 prefixes subnets announced to the customer. We know that almost everything works with Nat64 for ages, I think even Skype/teams might work with that . At the RIPE meeting (the bi yearly conference on internet governance in Europe), works in IPv6 Mostly (ref: https://ripe86.ripe.net/archives/video/1130/ ) . So yup, indeed, that might be an option for end-user networks. Now operators, CDN and webhosters shall ensure that they deploy IPv6 on their infra. But that’s another topic
I understand that, I know friends in Germany that also have terrible connection, even if they are in a decently well centered and big towns. Also remember that the 25G connection is only available in big towns.
Otoh, it’s a great proof of concept, and it seems that even with 25Gbps at home/work, (and some usage policy clearly stated by the ISPs), the supplier network will be rarely saturated, if they peer and interconnect " correctly " .
We got IoT and stuff going on now…we have actual end-user use case benefiting from their own subnet. I hope we don’t have to wait for everyone getting 25Gbps and hitting the NAT-wall of performance
I live in a 300k population northern German town and the local provider only speaks IPv4. Other one being Telekom with dual stack. Both aim to get 1Gbps fiber by 2030.
State of West Germany 2023
W. The ISPs in my country only route /64 to consumers, if they even support dual stack IPv6 in the first place.
The largest ISP in my country only started to transit from 6rd to native dual stack recently, and seem like they are routing /56 subnets so that’s a W.
Hopefully that puts pressure on the other ISPs to implement IPv6 properly if enough users complain about the /64 subnet…
It is truly ridiculous bordering on bizarre.
Took a bunch of back and forth, but since earlier this year, I have 250↓/50↑ Mbit, still on VDSL since running fiber from the buildings basement to my apartment was too much of an ask…
openwrt has this odhcpd, and it deals with multiple /64s on a private lan, and ISPs that will only do /64s.
The key feature for /64 only ISPs is “NDP relaying” or “ND proxy-ing” - I’m not sure what else is out there that does this.
If you have experience with arp proxies often used by vpn concentrators from back in the day when those were a thing … this is roughly a similar concept, but ND instead of ARP, because of IPv6.
At the end of the day, router still just forwards ipv6 back and forth and there’s no NAT needed and every device gets it’s own world routable /64 is it expects.
) .