Edit 1: On these tests I am using OpenWRT. On Pfsense I am getting around 300-400Mb/s avg.
You’ve not said what OS you’re using on the APU2, but I’m going to assume PFSense as you mentioned Teklager. Routing between interfaces on a single core you’re not going to get more than 500Mb/sec. This also matches up with my own APU2 performance. If you want a speed improvement, then you might need to consider OpenWRT or VyOS.
On those performance benchmarks I am using Openwrt. On Pfsense on the same machine it’s around 300Mb/s max.
What is interesting is that when using “router on a stick” configuration and using VLANS to break out a “WAN/LAN” on a single interface, the speed is great and as expected. I guess it has to do with how each NIC is handled and the load it puts on the system. Honestly if there was a way to tell the CPU to ignore trying to keep the load balanced and not over tax the CPU but just give me the best raw throughput there was available, I think that it would be much better than it is currently.
Throughput will depend on packet size, firewall rules, etc.
Just because an interface is gifs it does t necessarily mean the cpu can keep up
You are definitely being CPU limited in your testing. I’d bet if you had multiple streams coming from different ports you would be getting closer to gigabit performance in aggregate than you are seeing.
The APU2 is using an old low-power Jaguar core, which is like an 8 year old design. The age isn’t really an issue, because most Pentiums or Core i3s from that time period can do gigabit all day long, you’re talking about a 1GHz CPU that AMD really had designed for netbooks (remember when those were a thing??).
Sure it’s alot better than the Atom D525, but still it’s not fast…single threaded Passmark score of 412 is really what is hurting you.
A Pentium G2020T is from the same era, and it’s singlethreaded score is more than triple…
Since you have a switch around, have you tried using VLANs across 2 or 3 physical interfaces instead of doing router-on-a-stick?
Also, since you have a successful router-on-a-stick setup, can you try comparing perf profiles between a successful run and a broken run.
Yes I have tried both configuration setups. The setup in the OP at the top is not a ROaS config and it’s results are really poor.
I’ve moved on from using this device as a router. For gig fiber it’s not worth it to degrade the speed. I’ve set it up as a VPN device and backup machine that pushes backups off-site. This way it uses/needs only one interface and at one interface is able to push 1gbps both directions easy.
A slight tangent, but it’s worth noting that the Ivy Bridge Pentium being compared with the GX-412TC in the APU2 might have over 3x the single-thread performance, but it also has a TDP of 35W v.s. 6W. Regardless of differences in Intel or AMD’s TDP formula, no LGA1155 is going to fit in the same form factor as the APU2.
A more appropriate comparison would be the Celeron 1019Y with a TDP of 10W. It appears to have faster single-thread performance, but less than double, and loses in multi-thread performance from having half the cores and threads.
I’m personally running OpenWRT on an APU2D2 and I’m seeing similar performance between interfaces, and achieving gigabit throughput routed between VLANs on a single interface.
Hey man,
I totally understand your point and was not to be apples-to-apples but mainly illustrative of the point that the APU2’s time is over if you need gigabit routing. If an old desktop CPU can triple the performance 8 years ago, a more modern low power chip is also going to be substantially faster.
I’d be willing to bet that even the ARM-based SG-1100 far outperforms it. That thing has a 3 watt idle and it’s power supply is only 12v @ 2amp. It’s also smaller than the APU2
I totally understand your point and was not to be apples-to-apples but mainly illustrative of the point that the APU2’s time is over if you need gigabit routing.
Yeah, totally agree with you there. The APU2 is just not fast at all.
Just remembered, OpenWRT supports “software flow offloading” - this makes most packets from already established and routed TCP/UDP flows bypass routing and filtering code entirely. Depending on the device it can potentially double the throughput. It still goes through whatever qdisc code is there.
There’s also various experimental and hacky XDP / eBPF forwarding and NAT implementations, they bypass even that… I haven’t checked in a few months, but last I checked they weren’t worth considering seriously for any reliable/maintainable use case.
Also, FWIW, Raspberry Pi 4 can do gigabit (if you can get a non sucky gigabit USB adapter). So can various Odroid machines, and there’s also NanoPi R4S that can do it too. (Pi 4 doesn’t do AES in hardware, but wireguard is quick on it, wireguard is slower on others but they do AES in hardware if you want to stick external encrypted HDDs).
That’s good to know about the RPi4. Too bad that those are basically non existent in stock anywhere currently with shortages in supply and chips.
I decided to investigate this a bit more, since I was surprised myself to find that my APU2 was only routing at some 300-400 Mbps. I was sure that I had it routing at roughly gigabit line rate in the past.
Under a normal 1-directional iperf test, I saw that none of the 4 CPU cores came anywhere close to being fully utilised. That meant that there wasn’t a CPU bottleneck.
Then I ran a bidirectional iperf test, and I was seeing over 1700 Mbps combined throughput. How could a bidirectional test be faster than the sum of unidirectional tests?
I suspected some firmware settings, especially ones related to power management, might be to blame. I found that PCIe power management features were to blame. Disabling that, lo and behold, I once again had iperf throughput greater than 900 Mbps.
The APU2 may not be fast by modern standards, but it’s not that slow either. It can handle a single gigabit stream just fine. For multiple gigabit streams, one will have to pay attention to IRQ affinity and other tunables, but it is doable also.
Wow that was the “fix?” The upgraded Bios must set that to default. Did you see you’re speeds across interfaces for back to 900Mbs again after changing that setting?
Edit: I am able to confirm this under OpenWRT that turning off “PCIe power management features” brings the speed back near gigabit routing between interfaces. I wonder what that setting is used for and why it’s causing such a big dip in performance. @gnustomp What OS are you running on yours? Have you done tests with both FreeBSD (Pfsense or OPNsene) and Linux (OpenWRT)?
I’m using OpenWrt. According to https://review.coreboot.org/c/coreboot/+/39704 it enables ASPM L0s and L1, CommClk and ClockPM. I haven’t looked into which of these settings causes the performance degradation and why yet. To check whether they are supported or enabled, run lspci -vv and look at the LnkCtl and LnkSta fields (at least on Linux, not sure about BSD).
It makes me wonder now if Pfsense sees the same improvement. I had given up on this box being a low powered router in my setup when I saw those speeds but if this fix works then I might be back in business.
Edit: While the initial tests looked promising, once I added in the EAP Authentication needed for my fiber connection line, speeds dropped again. Turns out netgraph under Pfsense is severely limited by single core performance which the APU2 lacks. 
Lesson learned. I know people above mentioned it but at least it’s good to be able to test it out. My current setup with my intel G4560 has good enough single threaded speeds to handle it no sweat.
It’s funny to see how the PfSense performance question is still alive (or was) .
We (at work) moved most of our APUs 2 to OPNSense during the past years, and really didn’t had any performances problems on simple nat/routing.
As I’ve been running a simple debian with iptables on my APU2 (before decommissioning it recently, planning my line upgrade) for years, I can confirm that 700-800Mbps toward Internet from an internal machine isn’t a problem. As soon as you try to do openvpn, you’ll see lower performances, and didn’t tried with wireguard recently. Note that I’m running 1Gbps FTTH symetrical, and that I had control on decent Iperf servers connected at 10Gbps with a know path on the wild internet. Hope this confirmation helps.
