I have 2 sites that each have a 10G fiber internet connection. So far I am working at roughly 2.5 to 3 Gbps. The routers on there own tend to get about 6.5 Gbps to 8 Gbps on there own (usage dependent).
I suspect that I am being limited by how fast the CPU can encrypt, tunnel, decrypt and route my traffic. (I am not a network engineer so I apologize if I am using the wrong terms.)
Both sites are running PFSense
CPU Info @ Site A
Intel(R) Core™ i7-6700 CPU @ 3.40GHz
8 CPUs: 1 package(s) x 4 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
CPU Info @ Site B
Intel(R) Xeon(R) E-2278G CPU @ 3.40GHz
16 CPUs: 1 package(s) x 8 core(s) x 2 hardware threads
AES-NI CPU Crypto: Yes (active)
Beyond age of the chips I am wondering if a higher base clocked CPU will help. I am also not sure if PFSense will take advantage of boost clock and due to consistent usage I would rather have a higher base and not worry as much about boost. Correct me if my logic is wrong here. This config has run fine on a 1G symmetrical but now that 10G is finally on both ends its upgrade time.
To this end I am considering the Intel® Xeon® W-1250P Processor with its 6c/12t and a 4.1 GHz base clock and a Super Micro X12SCZ-TLN4F (I picked this specific board because of its integrated 2x 10G ports and 2x 1G ports (I still run some isolated LANs for VMs and Dev / Test.)
For anyone curious about how / why I use so much data its mainly due to VMs and data replication.
I am currently using WireGuard for the S2S links.
It’s my first time in the forums here so please be kind with feedback on my post.
Assuming this is a commercial setup, consider switching to AMD EPYC: more cores/threads to process your network connections. You can get used 1st gen EPYC CPU’s for not a lot (Aliexpress!) and suitable mainboards are also not too expensive. Add coolers and RAM and your basic system is ready. Add some SFP+ NIC’s and you’d be able to connect directly to the ISP fibre, at least in theory.
If you are doing a site to site on OpenVPN then you will usually be core speed limited since it uses a single thread. Some of the traffic can be accelerated via the AES-NI part of the CPU if it supports the encryption you are using in your config. If you are on OpenVPN, consider switching to Wireguard as it is generally faster. But honestly 2.5gb+ is really good already for VPN throughput
EniGmA1987 Thank you for your feedback as noted near the bottom I am currently using WireGuard. I can confirm that it did increase speed. I was peaking out around 800-1000 Mbps on the OpenVPN tunnel and on the WireGuard tunnel I am getting the current 2.5 to 3 Gbps. I however did play around with other settings in that process so I can not confirm if the change in tunneling exclusively affected the speed or not. AES-NI is enabled and this had a very significant increase on speed.
Dutch_Master, thank you for the feedback. I did do some limited testing with an AMD system and to be honest the results were disappointing. Maybe I had something incorrectly configured? I rebuilt PFSense from scratch and was getting around 300-500 Mbps. Couldn’t even break a gig.
Is more cores less speed or less cores more speed better for high throughput? Also I am using an X520-2 on each side for WAN and LAN. The other ports are for isolated LANs and testing.
Are you able to test without the internet link in between?
How are you testing? If using iperf, are you using a single thread?
Is any if your internet links connecting using pppoe?
Have you tried changing the MTU on both sides?
Is the max speed achieve on both directions?
Can you try using a Linux based router instead of offense? Vyos would be easy if you are a cli guy…
Ah ok, sorry I missed that part. Wireguard does multi-core encryption so I would bet you would benefit from higher core count CPUs as far as throughput goes. You are likely limited on the encrypt/decrypt of the traffic.
You may be able to test how much the cores affect your speed. The 7600K system seems like it is maybe custom built? If so you should go into the bios and disable 2 of the cores. Check your speed and see if it dropped a significant amount. That would let you know how much adding more cores would possibly help you out.
I’m also curious to see what VyOS would do in this situation. I’m in the process of selecting server hardware with a flexible budget under $10k per server for terminating wireguard vpn connections from a number of 1Gb clients with 10-50 home internet users behind each. We’re using vyos on some older xeon hardware at the moment.