I have a Wireguard tunnel between my homelab pfSense and a VPS. The VPS Wireguard is the gateway for the lab subnet on pfSense, and in conjunction with some iptables rules on the VPS, all my traffic exits and enters from the VPS. This works perfectly with no DNS leaks (to my slight surprise). This affords me a nice clean public IP, as opposed to my blacklisted residential address.
The problem I’m having is that the speed is adequate, but ONLY adequate. It saturates my home upload (100mbps), which I’m thankful for, but I’m only getting about 150mbps download. My home download is gigabit, and the VPS is advertised as synchronous gigabit - generally i’ve seen about 800mbps - which would be fine IF I could actually achieve that.
But I can’t. iperf3 in every direction shows an abundance of bandwidth, I’ve bought a larger VPS with 4 cores, I’ve tried every tuneable under the sun (from MTU to GRO), but the wireguard tunnel just refuses to run any faster than 150mbps. I’m not CPU bound on either end.
Correct, over wireguard 150. I’ve tried iperf on running pfSense ,and iperf on a VM (i.e. with the WG traffic routed through the firewall) - both the same. Iperf clean (even UDP) is well in the clear, up around 800-1000.
I’ve also tried connecting to the VPS wireguard on my laptop on a different, commercial internet connection, and I still can’t surpass 150. This suggests to me that it’s definitely a problem on the VPS end and not my ISP/homelab setup.
As far as MTU, I’ve tried 1384, 1420 and 1440 (i’m doing IPV4 only), but I can’t see any difference. Perhaps there’s a sweet spot that I just haven’t found? Maybe I’ll run a script to test all the MTUs.
Probably some cryptographic offload bottleneck. You wont see it as CPU load because it is offloaded to a hardware accelerator, but that accelerator in the CPU is probably bogged down. Or just not passing right through the virtual machine.
Wireguard is all in software, chacha20 mostly, and it’s UDP
I wonder if @scruffyte ran 1 connection iperf3 over UDP, what would he manage to pull
I don’t know FreeBSD networking stack that well, on Linux you can “cheat” and rewrite packets on both ends to look like something else, to try and confuse middle boxes and firewalls in between.
That’d be my next stop if this were Linux, it’s easy to do.
Next thing, I’d look at trying to figure out how to increase buffer sizes and parallelism, that’d require tuning the network stack parameters.
I’ve just stubled across a paper from Tuebingen University, and according to their results Wireguard performs significantly worse in a virtualised environment with no cpu pinning. With pinning it peforms about 40% better. Interestingly IPsec is not affected by this.
Perhaps my VPS is on a particularly crowded server and it’s causing wireguard to tank. I find that most likely as every other physical/hardware metric I can measure is in the green, leaving the software/CPU based/bound wireguard the only culprit.
I did try upgrading my VPS from 1 to 4 cores, which gave me a slight speed bump, but certainly not 3 additional cores worth. Very weird.
I think I’ll try IPsec and see if it makes a difference.
In case anyone stumbles across this in future, it was a VPS host issue. Whatever stack IONOS is running is crap, regardless of how many cores you add. I switched to Fasthosts and now I’m getting 400mbps.