Realtek 2.5Gbit nic transfer speed

Hi all, I’ve upgraded my home network to 2.5GBit/s but have some performance problems.

PC and NAS have RTL8125 network cards. For the PC, it’s just the default integrated nic, in the nas it’s connected via pcie. Between them is an opnsense box with a RTL8125B 4-Port nic in bridge mode. Opnsense CPU is AMD GX 415GA, TCP Offload Engine has been both disabled and enabled during testing in System → Settings → Tunables.
Using kernel module r8169, which is what both systems default to, the transfer speed between nas and pc is at best around 140MB/s. Installing kernel module r8125, which is what realtek recommends, from either their website or from one of the github sites, has no impact on transfer speed at all. Use of module r8125 has been verified on both devices via “lspci -k”. Opnsense has firmware plugin os-realtek-re installed. Speed has been tested via rsync file transfers and iperf, the latter of which reported only 1.18Gbit/s.

PC is running ubuntu 24.04 with kernel 6.8.0-51-generic.
NAS is running ubuntu server 24.04 with kernel 6.8.0-51-generic.

I was thinking that perhaps the CPU in the opnsense box could be bottlenecking transfers, but with TCP Offloading that shouldn’t be the case, right? Help in finding and fixing the problem would be much appreciated.

To start eliminating variables I would try first connecting PC and NAS directly to each other and running iperf in that configuration. If you’ll get near 2.5g - then the problem is in the switch in the middle

1 Like

Good idea, iperf reports 2.35Gbit/s when connecting them directly. Far from perfect, but close enough right now. Question is, is there a way to “fix” the speed when having the opnsense box connected between them? It’s nic should be able to handle 2.5Gbit/s as well and if I understand it correctly (I might not), TCP Offloading should take the CPU out of the equation.
If all else fails I could just return the 4 Port nic in the opnsense box and get a 2.5Gbit Switch instead, but I’d much prefer keeping the current bridged configuration.

Edit: Since the issue is with the opnsense machine, I’ve changed the flair to BSD.
Opnsense details:
OPNsense 24.7.11_2-amd64
FreeBSD 14.1-RELEASE-p6
realtek plugin:
os-realtek-re (installed) 1.0 409B 3 OPNsense Realtek re(4) vendor driver

What is the CPU load on the opnsense during the test?

highest I’ve seen it go (while transfering a 20GB file to the nas) was load of 2.21. That’s high for a four-core system, but there’s also some room left, right?

I’m not sure you can do routing/firewalling utilizing multiple cores. I’d say a single core performance is of significance here. Also - i’d leave the tests with files for now, just to eliminate more variables. Afaik, opnsense should have iperf package too - what would be the result between pc and firewall?

Took me a moment to figure out that the iperf package for opnsense is called iperf3 and that the linux client needs to also have iperf3 installed, not iperf without number, just in case anyone else will stumble upon this thread in the future.

iperf3 between firewall and pc:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 539 MBytes 452 Mbits/sec 0 sender
[ 5] 0.00-10.00 sec 538 MBytes 451 Mbits/sec receiver

this is shockingly slow, but the result doesn’t really make sense given that pc to nas is much faster.

Edit: using iperf3 between nas and pc for the sake of consistency:
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.39 GBytes 1.19 Gbits/sec 0 sender
[ 5] 0.00-10.01 sec 1.38 GBytes 1.19 Gbits/sec receiver

Try running it with multiple tcp streams (if i remember correctly that’s -P option). But all looks like the cpu is the bottleneck on the firewall

running with -P 4:
[ 5] 9.00-10.00 sec 14.0 MBytes 117 Mbits/sec 0 322 KBytes
[ 7] 9.00-10.00 sec 13.9 MBytes 116 Mbits/sec 0 318 KBytes
[ 9] 9.00-10.00 sec 13.9 MBytes 116 Mbits/sec 0 318 KBytes
[ 11] 9.00-10.00 sec 12.1 MBytes 102 Mbits/sec 0 298 KBytes
[SUM] 9.00-10.00 sec 53.9 MBytes 452 Mbits/sec

I guess I should’ve expected this, the GX 415GA is quite old.
Thanks for helping me pinpoint this, I’ll get an external 4x 2.5G switch and return the 4 port realtek nic in the opnsense.

Yeah, a dedicated switch is probably a more appropriate way to approach 2.5g in your LAN, if your budget allows it of course. Regarding TCP offloading - it doesn’t mean that CPU is not participating in traffic handling at all, but only certain functions are executed by the NIC card processor (for example computing checksums for the packets). Still the firewalling decisions, passing though iptables (if it was a linux box), etc. are made on the CPU

1 Like

Don’t use Realtek NICs and the GX 415 CPU is really old and slow which very likely doesn’t help your cause even if you had a beefier CPU.