First iperf3 test is 20% of link speed, subsequent tests are full speed

Hardware Hub Networking
1

Network layout:

image
image1362×1321 65.9 KB

The problem:
Sometimes, iperf3 test #1 on the 10GbE network results in barely over 1GbE speeds. Subsequent tests result in full 10GbE speeds.

iperf3 Output

root@truenas[~]# iperf3 -s

Server listening on 5201 (test #1)

Accepted connection from 10.0.0.2, port 54058
[ 5] local 10.0.0.1 port 5201 connected to 10.0.0.2 port 54059
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 172 MBytes 1.44 Gbits/sec
[ 5] 1.00-2.00 sec 175 MBytes 1.47 Gbits/sec
[ 5] 2.00-3.00 sec 176 MBytes 1.47 Gbits/sec
[ 5] 3.00-4.00 sec 176 MBytes 1.48 Gbits/sec
[ 5] 4.00-5.00 sec 176 MBytes 1.48 Gbits/sec
[ 5] 5.00-6.00 sec 177 MBytes 1.49 Gbits/sec
[ 5] 6.00-7.00 sec 176 MBytes 1.48 Gbits/sec
[ 5] 7.00-8.00 sec 175 MBytes 1.47 Gbits/sec
[ 5] 8.00-9.00 sec 175 MBytes 1.47 Gbits/sec
[ 5] 9.00-10.00 sec 176 MBytes 1.47 Gbits/sec
[ 5] 10.00-10.01 sec 2.34 MBytes 1.48 Gbits/sec

[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 1.72 GBytes 1.47 Gbits/sec receiver

Server listening on 5201 (test #2)

Accepted connection from 10.0.0.2, port 54060
[ 5] local 10.0.0.1 port 5201 connected to 10.0.0.2 port 54061
[ ID] Interval Transfer Bitrate
[ 5] 0.00-1.00 sec 1.08 GBytes 9.32 Gbits/sec
[ 5] 1.00-2.00 sec 1.10 GBytes 9.49 Gbits/sec
[ 5] 2.00-3.00 sec 1.11 GBytes 9.49 Gbits/sec
[ 5] 3.00-4.00 sec 1.11 GBytes 9.49 Gbits/sec
[ 5] 4.00-5.00 sec 1.11 GBytes 9.49 Gbits/sec
[ 5] 5.00-6.00 sec 1.11 GBytes 9.49 Gbits/sec
[ 5] 6.00-7.00 sec 1.10 GBytes 9.49 Gbits/sec
[ 5] 7.00-8.00 sec 1.10 GBytes 9.49 Gbits/sec
[ 5] 8.00-9.00 sec 1.11 GBytes 9.49 Gbits/sec
[ 5] 9.00-10.00 sec 1.11 GBytes 9.49 Gbits/sec
[ 5] 10.00-10.01 sec 10.1 MBytes 9.49 Gbits/sec

[ ID] Interval Transfer Bitrate
[ 5] 0.00-10.01 sec 11.0 GBytes 9.48 Gbits/sec receiver

Server listening on 5201 (test #3)

Exiting and re-launching the iperf3 server doesn’t trigger the single slow test. Neither does rebooting the client. Which to say it has happened multiple times, but isn’t (yet) repeatable beyond the anecdotal first test of the day. A iperf3 speed test over the second port on the same card on the 1GbE network does not prevent the first slower test on the 10GbE network.

As far as I have been able to tell, samba network transfer speeds aren’t changed between multiple runs nor before/after the NIC ‘wakes up’ with an iperf3 test.

My question:

Is this just a thing that happens?

1 Like
2

disable pcie power saving
keep it live all the time

Network cards will default to lower states, but cannot change under load due to fear of packet loss.

2 Likes
3

Try adding -t 120 to make iperf3 run for a couple minutes.

1 Like
4

Doesnt EEE (Energy Efficient Ethernet) downgrade the link speed to save power and increase it back up when it detects that you need the bandwidth?
PCIE energy saving does the same.

Make sure you have both those options off in the bios and driver.

2 Likes
5

I’ve been looking into this, starting with my W10 machine in the ‘Advanced’ tab of the NIC properties menu, such options aren’t present. They ARE present for my onboard NICs, just not the x540-t card.

I’ll investigate BIOS next for PCIE power saving options.

I’ll cross the bridge of changing that in Truenas Core when I get there :dizzy_face:

EDIT: I finally saw the ‘Power Management’ tab, unselecting ‘Allow the competer to turn off this device to save power’ eliminated this morning’s slow iperf3 test. I’ll test again tomorrow :+1:

image
image857×952 32.1 KB

3 Likes
6

The same behavior has returned today. Seems like it is some kind of idle power state thing, so I’ll keep looking for something like that.