OPNSense 10Gbps iPerf3 Tests -- Packet Loss on "receiver"

johntdavis · June 25, 2023, 3:37am

I should say up front that I’m very new to 10Gbps networking and not used to using iperf3 in UDP mode, but I tried it since that’s whats recommended with 10Gbps speed tests.

I’m getting these results. A few questions:
(1) I don’t understand what a packet loss on receiver (but not sender) means in this context;
(2) Whatever causes (1), why is it worse in -R (reverse) mode?
(3) Is this something I actually need to worry about, or is it normal?

System is a Dell Optiplex 5040.

Intel(R) Core™ i5-6500 CPU @ 3.20GHz (4 cores, 4 threads) with 16 GB of DDR3 RAM;
The LAN NIC is a 10Gbps Intel X520-DA2.
Client and OPNSense are both set to MTU 9000.

Here are the test results:

% iperf3 -c 10.10.100.1  -u -i 1 -t 30 -b 10G -R
Connecting to host 10.10.100.1, port 5201
Reverse mode, remote host 10.10.100.1 is sending
[  5] local 10.10.100.130 port 57660 connected to 10.10.100.1 port 5201
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-1.00   sec  1.13 GBytes  9.74 Gbits/sec  0.013 ms  2316/138313 (1.7%)  
[  5]   1.00-2.00   sec  1.14 GBytes  9.76 Gbits/sec  0.007 ms  2360/138672 (1.7%)  
[  5]   2.00-3.00   sec  1.13 GBytes  9.75 Gbits/sec  0.013 ms  13581/149759 (9.1%)  
[  5]   3.00-4.00   sec  1.14 GBytes  9.76 Gbits/sec  0.008 ms  15122/151470 (10%)  
[  5]   4.00-5.00   sec  1.14 GBytes  9.76 Gbits/sec  0.015 ms  13145/149444 (8.8%)  
[  5]   5.00-6.00   sec  1.14 GBytes  9.76 Gbits/sec  0.010 ms  13621/149930 (9.1%)  
[  5]   6.00-7.00   sec  1.14 GBytes  9.76 Gbits/sec  0.008 ms  13310/149680 (8.9%)  
[  5]   7.00-8.00   sec  1.14 GBytes  9.75 Gbits/sec  0.013 ms  8859/145097 (6.1%)  
[  5]   8.00-9.00   sec  1.14 GBytes  9.75 Gbits/sec  0.009 ms  14099/150322 (9.4%)  
[  5]   9.00-10.00  sec  1.14 GBytes  9.76 Gbits/sec  0.009 ms  14747/151078 (9.8%)  
[  5]  10.00-11.00  sec  1.14 GBytes  9.76 Gbits/sec  0.006 ms  10456/146771 (7.1%)  
[  5]  11.00-12.00  sec  1.14 GBytes  9.76 Gbits/sec  0.014 ms  14305/150604 (9.5%)  
[  5]  12.00-13.00  sec  1.14 GBytes  9.75 Gbits/sec  0.011 ms  12657/148913 (8.5%)  
[  5]  13.00-14.00  sec  1.14 GBytes  9.76 Gbits/sec  0.007 ms  14013/150297 (9.3%)  
[  5]  14.00-15.00  sec  1.14 GBytes  9.76 Gbits/sec  0.010 ms  11611/147921 (7.8%)  
[  5]  15.00-16.00  sec  1.14 GBytes  9.76 Gbits/sec  0.006 ms  12737/149092 (8.5%)  
[  5]  16.00-17.00  sec  1.14 GBytes  9.76 Gbits/sec  0.015 ms  11856/148154 (8%)  
[  5]  17.00-18.00  sec  1.14 GBytes  9.75 Gbits/sec  0.012 ms  12527/148769 (8.4%)  
[  5]  18.00-19.00  sec  1.14 GBytes  9.76 Gbits/sec  0.009 ms  12571/148931 (8.4%)  
[  5]  19.00-20.00  sec  1.14 GBytes  9.75 Gbits/sec  0.008 ms  12030/148257 (8.1%)  
[  5]  20.00-21.00  sec  1.14 GBytes  9.76 Gbits/sec  0.014 ms  13026/149355 (8.7%)  
[  5]  21.00-22.00  sec  1.14 GBytes  9.76 Gbits/sec  0.008 ms  10384/146685 (7.1%)  
[  5]  22.00-23.00  sec  1.14 GBytes  9.76 Gbits/sec  0.011 ms  12672/148974 (8.5%)  
[  5]  23.00-24.00  sec  1.14 GBytes  9.76 Gbits/sec  0.008 ms  12339/148628 (8.3%)  
[  5]  24.00-25.00  sec  1.14 GBytes  9.76 Gbits/sec  0.008 ms  14110/150410 (9.4%)  
[  5]  25.00-26.00  sec  1.14 GBytes  9.76 Gbits/sec  0.011 ms  12614/148907 (8.5%)  
[  5]  26.00-27.00  sec  1.14 GBytes  9.76 Gbits/sec  0.010 ms  13323/149636 (8.9%)  
[  5]  27.00-28.00  sec  1.14 GBytes  9.76 Gbits/sec  0.013 ms  11472/147820 (7.8%)  
[  5]  28.00-29.00  sec  1.14 GBytes  9.75 Gbits/sec  0.008 ms  10721/146932 (7.3%)  
[  5]  29.00-30.00  sec  1.14 GBytes  9.76 Gbits/sec  0.012 ms  12844/149161 (8.6%)  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-30.00  sec  34.7 GBytes  9.93 Gbits/sec  0.000 ms  0/4450877 (0%)  sender
[SUM]  0.0-30.0 sec  17 datagrams received out-of-order
[  5]   0.00-30.00  sec  34.1 GBytes  9.76 Gbits/sec  0.012 ms  359428/4447982 (8.1%)  receiver

% iperf3 -c 10.10.100.1  -u -i 1 -t 60 -b 10G
Connecting to host 10.10.100.1, port 5201
[  5] local 10.10.100.130 port 62816 connected to 10.10.100.1 port 5201
[ ID] Interval           Transfer     Bitrate         Total Datagrams
[  5]   0.00-1.00   sec  1.16 GBytes  9.99 Gbits/sec  139634  
[  5]   1.00-2.00   sec  1.16 GBytes  9.96 Gbits/sec  139260  
[  5]   2.00-3.00   sec  1.16 GBytes  9.93 Gbits/sec  138837  
[  5]   3.00-4.00   sec  1.15 GBytes  9.86 Gbits/sec  137970  
[  5]   4.00-5.00   sec  1.16 GBytes  9.99 Gbits/sec  139768  
[  5]   5.00-6.00   sec  1.15 GBytes  9.91 Gbits/sec  138642  
[  5]   6.00-7.00   sec  1.15 GBytes  9.87 Gbits/sec  138065  
[  5]   7.00-8.00   sec  1.16 GBytes  9.99 Gbits/sec  139765  
[  5]   8.00-9.00   sec  1.15 GBytes  9.92 Gbits/sec  138687  
[  5]   9.00-10.00  sec  1.15 GBytes  9.88 Gbits/sec  138187  
[  5]  10.00-11.00  sec  1.16 GBytes  9.98 Gbits/sec  139553  
[  5]  11.00-12.00  sec  1.15 GBytes  9.92 Gbits/sec  138708  
[  5]  12.00-13.00  sec  1.15 GBytes  9.89 Gbits/sec  138435  
[  5]  13.00-14.00  sec  1.16 GBytes  9.96 Gbits/sec  139277  
[  5]  14.00-15.00  sec  1.15 GBytes  9.92 Gbits/sec  138711  
[  5]  15.00-16.00  sec  1.16 GBytes  9.93 Gbits/sec  138913  
[  5]  16.00-17.00  sec  1.16 GBytes  9.92 Gbits/sec  138782  
[  5]  17.00-18.00  sec  1.16 GBytes  9.93 Gbits/sec  138916  
[  5]  18.00-19.00  sec  1.15 GBytes  9.92 Gbits/sec  138767  
[  5]  19.00-20.00  sec  1.16 GBytes  9.93 Gbits/sec  138952  
[  5]  20.00-21.00  sec  1.15 GBytes  9.91 Gbits/sec  138671  
[  5]  21.00-22.00  sec  1.16 GBytes  9.93 Gbits/sec  138903  
[  5]  22.00-23.00  sec  1.15 GBytes  9.92 Gbits/sec  138704  
[  5]  23.00-24.00  sec  1.16 GBytes  9.93 Gbits/sec  138929  
[  5]  24.00-25.00  sec  1.16 GBytes  9.92 Gbits/sec  138821  
[  5]  25.00-26.00  sec  1.16 GBytes  9.93 Gbits/sec  138840  
[  5]  26.00-27.00  sec  1.16 GBytes  9.92 Gbits/sec  138795  
[  5]  27.00-28.00  sec  1.16 GBytes  9.93 Gbits/sec  138860  
[  5]  28.00-29.00  sec  1.16 GBytes  9.93 Gbits/sec  138854  
[  5]  29.00-30.00  sec  1.15 GBytes  9.92 Gbits/sec  138734  
[  5]  30.00-31.00  sec  1.16 GBytes  9.93 Gbits/sec  138902  
[  5]  31.00-32.00  sec  1.15 GBytes  9.92 Gbits/sec  138748  
[  5]  32.00-33.00  sec  1.16 GBytes  9.93 Gbits/sec  138874  
[  5]  33.00-34.00  sec  1.16 GBytes  9.92 Gbits/sec  138831  
[  5]  34.00-35.00  sec  1.16 GBytes  9.93 Gbits/sec  138892  
[  5]  35.00-36.00  sec  1.15 GBytes  9.92 Gbits/sec  138776  
[  5]  36.00-37.00  sec  1.16 GBytes  9.93 Gbits/sec  138886  
[  5]  37.00-38.00  sec  1.15 GBytes  9.92 Gbits/sec  138702  
[  5]  38.00-39.00  sec  1.16 GBytes  9.93 Gbits/sec  138921  
[  5]  39.00-40.00  sec  1.16 GBytes  9.93 Gbits/sec  138845  
[  5]  40.00-41.00  sec  1.16 GBytes  9.92 Gbits/sec  138819  
[  5]  41.00-42.00  sec  1.16 GBytes  9.92 Gbits/sec  138775  
[  5]  42.00-43.00  sec  1.16 GBytes  9.93 Gbits/sec  138880  
[  5]  43.00-44.00  sec  1.16 GBytes  9.92 Gbits/sec  138839  
[  5]  44.00-45.00  sec  1.15 GBytes  9.92 Gbits/sec  138684  
[  5]  45.00-46.00  sec  1.16 GBytes  9.93 Gbits/sec  138967  
[  5]  46.00-47.00  sec  1.16 GBytes  9.93 Gbits/sec  138875  
[  5]  47.00-48.00  sec  1.15 GBytes  9.92 Gbits/sec  138733  
[  5]  48.00-49.00  sec  1.16 GBytes  9.92 Gbits/sec  138787  
[  5]  49.00-50.00  sec  1.16 GBytes  9.93 Gbits/sec  138898  
[  5]  50.00-51.00  sec  1.16 GBytes  9.93 Gbits/sec  138833  
[  5]  51.00-52.00  sec  1.16 GBytes  9.93 Gbits/sec  138878  
[  5]  52.00-53.00  sec  1.15 GBytes  9.91 Gbits/sec  138656  
[  5]  53.00-54.00  sec  1.16 GBytes  9.93 Gbits/sec  138894  
[  5]  54.00-55.00  sec  1.16 GBytes  9.93 Gbits/sec  138935  
[  5]  55.00-56.00  sec  1.15 GBytes  9.92 Gbits/sec  138771  
[  5]  56.00-57.00  sec  1.16 GBytes  9.93 Gbits/sec  138819  
[  5]  57.00-58.00  sec  1.15 GBytes  9.91 Gbits/sec  138670  
[  5]  58.00-59.00  sec  1.16 GBytes  9.94 Gbits/sec  138950  
[  5]  59.00-60.00  sec  1.16 GBytes  9.93 Gbits/sec  138926  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Jitter    Lost/Total Datagrams
[  5]   0.00-60.00  sec  69.3 GBytes  9.93 Gbits/sec  0.000 ms  0/8330906 (0%)  sender
[  5]   0.00-60.00  sec  69.2 GBytes  9.90 Gbits/sec  0.006 ms  26911/8328876 (0.32%)  receiver

johntdavis · June 25, 2023, 3:38am

% iperf3 -c 10.10.100.1 -P 10
Connecting to host 10.10.100.1, port 5201
[  5] local 10.10.100.130 port 49662 connected to 10.10.100.1 port 5201
[  7] local 10.10.100.130 port 49663 connected to 10.10.100.1 port 5201
[  9] local 10.10.100.130 port 49664 connected to 10.10.100.1 port 5201
[ 11] local 10.10.100.130 port 49665 connected to 10.10.100.1 port 5201
[ 13] local 10.10.100.130 port 49666 connected to 10.10.100.1 port 5201
[ 15] local 10.10.100.130 port 49667 connected to 10.10.100.1 port 5201
[ 17] local 10.10.100.130 port 49668 connected to 10.10.100.1 port 5201
[ 19] local 10.10.100.130 port 49669 connected to 10.10.100.1 port 5201
[ 21] local 10.10.100.130 port 49670 connected to 10.10.100.1 port 5201
[ 23] local 10.10.100.130 port 49671 connected to 10.10.100.1 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   119 MBytes   999 Mbits/sec                  
[  7]   0.00-1.00   sec   119 MBytes   995 Mbits/sec                  
[  9]   0.00-1.00   sec   119 MBytes   994 Mbits/sec                  
[ 11]   0.00-1.00   sec   119 MBytes   995 Mbits/sec                  
[ 13]   0.00-1.00   sec   119 MBytes   995 Mbits/sec                  
[ 15]   0.00-1.00   sec   119 MBytes   995 Mbits/sec                  
[ 17]   0.00-1.00   sec   118 MBytes   993 Mbits/sec                  
[ 19]   0.00-1.00   sec   119 MBytes   999 Mbits/sec                  
[ 21]   0.00-1.00   sec   119 MBytes   995 Mbits/sec                  
[ 23]   0.00-1.00   sec   119 MBytes   995 Mbits/sec                  
[SUM]   0.00-1.00   sec  1.16 GBytes  9.95 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   118 MBytes   990 Mbits/sec                  
[  7]   1.00-2.00   sec   118 MBytes   990 Mbits/sec                  
[  9]   1.00-2.00   sec   118 MBytes   990 Mbits/sec                  
[ 11]   1.00-2.00   sec   118 MBytes   989 Mbits/sec                  
[ 13]   1.00-2.00   sec   118 MBytes   990 Mbits/sec                  
[ 15]   1.00-2.00   sec   118 MBytes   989 Mbits/sec                  
[ 17]   1.00-2.00   sec   118 MBytes   991 Mbits/sec                  
[ 19]   1.00-2.00   sec   118 MBytes   991 Mbits/sec                  
[ 21]   1.00-2.00   sec   118 MBytes   990 Mbits/sec                  
[ 23]   1.00-2.00   sec   118 MBytes   989 Mbits/sec                  
[SUM]   1.00-2.00   sec  1.15 GBytes  9.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   118 MBytes   991 Mbits/sec                  
[  7]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[  9]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[ 11]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[ 13]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[ 15]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[ 17]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[ 19]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[ 21]   2.00-3.00   sec   118 MBytes   989 Mbits/sec                  
[ 23]   2.00-3.00   sec   118 MBytes   990 Mbits/sec                  
[SUM]   2.00-3.00   sec  1.15 GBytes  9.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   116 MBytes   975 Mbits/sec                  
[  7]   3.00-4.00   sec   115 MBytes   963 Mbits/sec                  
[  9]   3.00-4.00   sec   115 MBytes   966 Mbits/sec                  
[ 11]   3.00-4.00   sec   115 MBytes   964 Mbits/sec                  
[ 13]   3.00-4.00   sec   115 MBytes   965 Mbits/sec                  
[ 15]   3.00-4.00   sec   115 MBytes   963 Mbits/sec                  
[ 17]   3.00-4.00   sec   115 MBytes   963 Mbits/sec                  
[ 19]   3.00-4.00   sec   116 MBytes   974 Mbits/sec                  
[ 21]   3.00-4.00   sec   115 MBytes   966 Mbits/sec                  
[ 23]   3.00-4.00   sec   115 MBytes   962 Mbits/sec                  
[SUM]   3.00-4.00   sec  1.12 GBytes  9.66 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   118 MBytes   989 Mbits/sec                  
[  7]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[  9]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[ 11]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[ 13]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[ 15]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[ 17]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[ 19]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[ 21]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[ 23]   4.00-5.00   sec   118 MBytes   990 Mbits/sec                  
[SUM]   4.00-5.00   sec  1.15 GBytes  9.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   116 MBytes   976 Mbits/sec                  
[  7]   5.00-6.00   sec   116 MBytes   969 Mbits/sec                  
[  9]   5.00-6.00   sec   115 MBytes   968 Mbits/sec                  
[ 11]   5.00-6.00   sec   115 MBytes   969 Mbits/sec                  
[ 13]   5.00-6.00   sec   115 MBytes   969 Mbits/sec                  
[ 15]   5.00-6.00   sec   115 MBytes   969 Mbits/sec                  
[ 17]   5.00-6.00   sec   116 MBytes   970 Mbits/sec                  
[ 19]   5.00-6.00   sec   116 MBytes   976 Mbits/sec                  
[ 21]   5.00-6.00   sec   116 MBytes   969 Mbits/sec                  
[ 23]   5.00-6.00   sec   115 MBytes   968 Mbits/sec                  
[SUM]   5.00-6.00   sec  1.13 GBytes  9.70 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   118 MBytes   992 Mbits/sec                  
[  7]   6.00-7.00   sec   118 MBytes   989 Mbits/sec                  
[  9]   6.00-7.00   sec   118 MBytes   989 Mbits/sec                  
[ 11]   6.00-7.00   sec   118 MBytes   990 Mbits/sec                  
[ 13]   6.00-7.00   sec   118 MBytes   989 Mbits/sec                  
[ 15]   6.00-7.00   sec   118 MBytes   989 Mbits/sec                  
[ 17]   6.00-7.00   sec   118 MBytes   989 Mbits/sec                  
[ 19]   6.00-7.00   sec   118 MBytes   991 Mbits/sec                  
[ 21]   6.00-7.00   sec   118 MBytes   990 Mbits/sec                  
[ 23]   6.00-7.00   sec   118 MBytes   990 Mbits/sec                  
[SUM]   6.00-7.00   sec  1.15 GBytes  9.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[  7]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[  9]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[ 11]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[ 13]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[ 15]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[ 17]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[ 19]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[ 21]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[ 23]   7.00-8.00   sec   118 MBytes   990 Mbits/sec                  
[SUM]   7.00-8.00   sec  1.15 GBytes  9.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   118 MBytes   990 Mbits/sec                  
[  7]   8.00-9.00   sec   118 MBytes   989 Mbits/sec                  
[  9]   8.00-9.00   sec   118 MBytes   990 Mbits/sec                  
[ 11]   8.00-9.00   sec   118 MBytes   990 Mbits/sec                  
[ 13]   8.00-9.00   sec   118 MBytes   990 Mbits/sec                  
[ 15]   8.00-9.00   sec   118 MBytes   990 Mbits/sec                  
[ 17]   8.00-9.00   sec   118 MBytes   990 Mbits/sec                  
[ 19]   8.00-9.00   sec   118 MBytes   992 Mbits/sec                  
[ 21]   8.00-9.00   sec   118 MBytes   989 Mbits/sec                  
[ 23]   8.00-9.00   sec   118 MBytes   990 Mbits/sec                  
[SUM]   8.00-9.00   sec  1.15 GBytes  9.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   118 MBytes   990 Mbits/sec                  
[  7]   9.00-10.00  sec   118 MBytes   989 Mbits/sec                  
[  9]   9.00-10.00  sec   118 MBytes   990 Mbits/sec                  
[ 11]   9.00-10.00  sec   118 MBytes   990 Mbits/sec                  
[ 13]   9.00-10.00  sec   118 MBytes   990 Mbits/sec                  
[ 15]   9.00-10.00  sec   118 MBytes   990 Mbits/sec                  
[ 17]   9.00-10.00  sec   118 MBytes   990 Mbits/sec                  
[ 19]   9.00-10.00  sec   118 MBytes   988 Mbits/sec                  
[ 21]   9.00-10.00  sec   118 MBytes   989 Mbits/sec                  
[ 23]   9.00-10.00  sec   118 MBytes   989 Mbits/sec                  
[SUM]   9.00-10.00  sec  1.15 GBytes  9.90 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.15 GBytes   988 Mbits/sec                  sender
[  5]   0.00-10.00  sec  1.15 GBytes   987 Mbits/sec                  receiver
[  7]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  sender
[  7]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[  9]   0.00-10.00  sec  1.15 GBytes   986 Mbits/sec                  sender
[  9]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.15 GBytes   986 Mbits/sec                  sender
[ 11]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[ 13]   0.00-10.00  sec  1.15 GBytes   986 Mbits/sec                  sender
[ 13]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[ 15]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  sender
[ 15]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[ 17]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  sender
[ 17]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[ 19]   0.00-10.00  sec  1.15 GBytes   988 Mbits/sec                  sender
[ 19]   0.00-10.00  sec  1.15 GBytes   987 Mbits/sec                  receiver
[ 21]   0.00-10.00  sec  1.15 GBytes   986 Mbits/sec                  sender
[ 21]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[ 23]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  sender
[ 23]   0.00-10.00  sec  1.15 GBytes   985 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  11.5 GBytes  9.86 Gbits/sec                  sender
[SUM]   0.00-10.00  sec  11.5 GBytes  9.85 Gbits/sec                  receiver

risk · June 26, 2023, 5:16am

UDP by design doesn’t have flow control and packets will routinely get dropped and not retransmitted when any of the buffers on source machine or destination machine are full and destination machine isn’t keeping up.

It’s very tedious to figure out where exactly this is happening the most, there’s usually counters you can check starting with ethtool and the driver in sysfs, but there’s a thousand places to look at.

Most workload those days is TCP, although QUIC which is UDP based is starting to appear - personally, I wouldn’t bother testing UDP performance.

johntdavis · June 27, 2023, 3:38pm

Thanks for the explanation and reassurance this is working as intended. I’m not super familiar with UDP, but it was recommended for more accurate perf3 tests in some articles I read so I decided to try it.

I’m really not getting different iperf3 results with UDP than I am with TCP, though, so at least on my network that’s not a thing to worry about. I’m getting 9+ Gbps of ludicrous speed, either way.