Testing 10GbE Subnet without PCIE slots

Intro

I want to connect my NAS to my plex server directly with a faster subnet. I’m currently using a 2.5GbE connection, but as we all know, the faster the better.

Neither have any PCIE slots available, or this would be straightforward..

My plex server is using an m.2 to 10GbE adapter:

My NAS is using a USBC to 10GbE adapter courtesy of wendell for testing:

I wanted to test this out after seeing Jeff Geerling’s article from earlier this year on USBC to 10GbE speeds; apparently it varies quite a lot depending on your setup!

iperf3 Data

Unraid NAS (big-boss) Plex/Tdarr server (little-boss)
10GbE NIC USB-C to 10GbE adapter (WisdPi) M.2 to PCIe 10GbE card
Chipset Realtek USB 10G controller (USB ID 0bda:815a), community Realtek drivers installed Aquantia AQC113 (10G)
10GbE IP 10.1.2.100/24 10.1.2.101/24
Link 10000 Mb/s, full duplex 10000 Mb/s, full duplex
iperf3 -c 10.1.2.101
Connecting to host 10.1.2.101, port 5201
[  5] local 10.1.2.100 port 55032 connected to 10.1.2.101 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   935 MBytes  7.83 Gbits/sec    0   1.03 MBytes
[  5]   1.00-2.00   sec   936 MBytes  7.85 Gbits/sec    0    877 KBytes
[  5]   2.00-3.00   sec   933 MBytes  7.83 Gbits/sec    0    829 KBytes
[  5]   3.00-4.00   sec   926 MBytes  7.77 Gbits/sec    0   1.43 MBytes
[  5]   4.00-5.00   sec   929 MBytes  7.79 Gbits/sec    0   1.56 MBytes
[  5]   5.00-6.00   sec   942 MBytes  7.89 Gbits/sec    0   1.21 MBytes
[  5]   6.00-7.00   sec   932 MBytes  7.82 Gbits/sec    0    888 KBytes
[  5]   7.00-8.00   sec   938 MBytes  7.87 Gbits/sec    0    913 KBytes
[  5]   8.00-9.00   sec   933 MBytes  7.82 Gbits/sec    0   1.17 MBytes
[  5]   9.00-10.00  sec   927 MBytes  7.77 Gbits/sec    0   5.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.11 GBytes  7.83 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  9.10 GBytes  7.81 Gbits/sec                  receiver

iperf Done.
iperf3 -c 10.1.2.100 -R
Connecting to host 10.1.2.100, port 5201
Reverse mode, remote host 10.1.2.100 is sending
[  5] local 10.1.2.101 port 58844 connected to 10.1.2.100 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   921 MBytes  7.72 Gbits/sec               
[  5]   1.00-2.00   sec   936 MBytes  7.85 Gbits/sec               
[  5]   2.00-3.00   sec   922 MBytes  7.74 Gbits/sec               
[  5]   3.00-4.00   sec   923 MBytes  7.74 Gbits/sec               
[  5]   4.00-5.00   sec   932 MBytes  7.83 Gbits/sec               
[  5]   5.00-6.00   sec   928 MBytes  7.78 Gbits/sec               
[  5]   6.00-7.00   sec   930 MBytes  7.80 Gbits/sec               
[  5]   7.00-8.00   sec   927 MBytes  7.78 Gbits/sec               
[  5]   8.00-9.00   sec   936 MBytes  7.85 Gbits/sec               
[  5]   9.00-10.00  sec   927 MBytes  7.77 Gbits/sec               
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.01  sec  9.09 GBytes  7.80 Gbits/sec    1             sender
[  5]   0.00-10.00  sec  9.08 GBytes  7.80 Gbits/sec                  receiver

iperf Done.
iperf3 -c 10.1.2.101
Connecting to host 10.1.2.101, port 5201
[  5] local 10.1.2.100 port 55032 connected to 10.1.2.101 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   935 MBytes  7.83 Gbits/sec    0   1.03 MBytes
[  5]   1.00-2.00   sec   936 MBytes  7.85 Gbits/sec    0    877 KBytes
[  5]   2.00-3.00   sec   933 MBytes  7.83 Gbits/sec    0    829 KBytes
[  5]   3.00-4.00   sec   926 MBytes  7.77 Gbits/sec    0   1.43 MBytes
[  5]   4.00-5.00   sec   929 MBytes  7.79 Gbits/sec    0   1.56 MBytes
[  5]   5.00-6.00   sec   942 MBytes  7.89 Gbits/sec    0   1.21 MBytes
[  5]   6.00-7.00   sec   932 MBytes  7.82 Gbits/sec    0    888 KBytes
[  5]   7.00-8.00   sec   938 MBytes  7.87 Gbits/sec    0    913 KBytes
[  5]   8.00-9.00   sec   933 MBytes  7.82 Gbits/sec    0   1.17 MBytes
[  5]   9.00-10.00  sec   927 MBytes  7.77 Gbits/sec    0   5.66 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  9.11 GBytes  7.83 Gbits/sec    0            sender
[  5]   0.00-10.00  sec  9.10 GBytes  7.81 Gbits/sec                  receiver

iperf Done.
iperf3 -c 10.1.2.100 -P 4
Connecting to host 10.1.2.100, port 5201               [  5] local 10.1.2.101 port 33862 connected to 10.1.2.100 port 5201                                           [  7] local 10.1.2.101 port 33878 connected to 10.1.2.100 port 5201                                           [  9] local 10.1.2.101 port 33890 connected to 10.1.2.100 port 5201                                           [ 11] local 10.1.2.101 port 33900 connected to 10.1.2.100 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   136 MBytes  1.14 Gbits/sec    0    580 KBytes                                       [  7]   0.00-1.00   sec   163 MBytes  1.36 Gbits/sec    0    584 KBytes
[  9]   0.00-1.00   sec   420 MBytes  3.52 Gbits/sec    0   1.13 MBytes
[ 11]   0.00-1.00   sec   130 MBytes  1.09 Gbits/sec    0    479 KBytes
[SUM]   0.00-1.00   sec   848 MBytes  7.11 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec   150 MBytes  1.26 Gbits/sec    0    580 KBytes                                       [  7]   1.00-2.00   sec   150 MBytes  1.26 Gbits/sec    0    615 KBytes
[  9]   1.00-2.00   sec   418 MBytes  3.51 Gbits/sec    0   1.25 MBytes
[ 11]   1.00-2.00   sec   119 MBytes  1.00 Gbits/sec    0    479 KBytes
[SUM]   1.00-2.00   sec   837 MBytes  7.02 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec   149 MBytes  1.25 Gbits/sec    0    611 KBytes
[  7]   2.00-3.00   sec   149 MBytes  1.25 Gbits/sec    0    615 KBytes
[  9]   2.00-3.00   sec   419 MBytes  3.51 Gbits/sec    0   1.25 MBytes
[ 11]   2.00-3.00   sec   120 MBytes  1.01 Gbits/sec    0    479 KBytes
[SUM]   2.00-3.00   sec   838 MBytes  7.03 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec   149 MBytes  1.25 Gbits/sec    0    611 KBytes
[  7]   3.00-4.00   sec   149 MBytes  1.25 Gbits/sec    0    615 KBytes
[  9]   3.00-4.00   sec   418 MBytes  3.51 Gbits/sec    0   1.31 MBytes
[ 11]   3.00-4.00   sec   119 MBytes  1.00 Gbits/sec    0    479 KBytes
[SUM]   3.00-4.00   sec   836 MBytes  7.01 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec   150 MBytes  1.26 Gbits/sec    0    639 KBytes
[  7]   4.00-5.00   sec   149 MBytes  1.25 Gbits/sec    0    615 KBytes
[  9]   4.00-5.00   sec   419 MBytes  3.51 Gbits/sec    0   1.31 MBytes
[ 11]   4.00-5.00   sec   119 MBytes  1.00 Gbits/sec    0    479 KBytes
[SUM]   4.00-5.00   sec   838 MBytes  7.03 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec   154 MBytes  1.29 Gbits/sec    0    669 KBytes
[  7]   5.00-6.00   sec   148 MBytes  1.24 Gbits/sec    0    615 KBytes
[  9]   5.00-6.00   sec   418 MBytes  3.51 Gbits/sec    0   1.31 MBytes
[ 11]   5.00-6.00   sec   117 MBytes   985 Mbits/sec    0    479 KBytes
[SUM]   5.00-6.00   sec   838 MBytes  7.03 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec   150 MBytes  1.26 Gbits/sec    0    906 KBytes
[  7]   6.00-7.00   sec   129 MBytes  1.08 Gbits/sec    0    773 KBytes
[  9]   6.00-7.00   sec   410 MBytes  3.44 Gbits/sec    0   1.60 MBytes
[ 11]   6.00-7.00   sec   147 MBytes  1.23 Gbits/sec    1    946 KBytes
[SUM]   6.00-7.00   sec   836 MBytes  7.02 Gbits/sec    1
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec   145 MBytes  1.22 Gbits/sec    0    906 KBytes
[  7]   7.00-8.00   sec   122 MBytes  1.02 Gbits/sec    0    773 KBytes
[  9]   7.00-8.00   sec   419 MBytes  3.51 Gbits/sec    0   1.60 MBytes
[ 11]   7.00-8.00   sec   150 MBytes  1.26 Gbits/sec    0    946 KBytes
[SUM]   7.00-8.00   sec   836 MBytes  7.01 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec   147 MBytes  1.23 Gbits/sec    0    906 KBytes
[  7]   8.00-9.00   sec   120 MBytes  1.01 Gbits/sec    0    773 KBytes
[  9]   8.00-9.00   sec   418 MBytes  3.51 Gbits/sec    0   1.60 MBytes
[ 11]   8.00-9.00   sec   151 MBytes  1.27 Gbits/sec    0    946 KBytes
[SUM]   8.00-9.00   sec   837 MBytes  7.03 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec   145 MBytes  1.22 Gbits/sec    0    906 KBytes
[  7]   9.00-10.00  sec   122 MBytes  1.02 Gbits/sec    0    773 KBytes
[  9]   9.00-10.00  sec   419 MBytes  3.51 Gbits/sec    0   1.60 MBytes
[ 11]   9.00-10.00  sec   150 MBytes  1.26 Gbits/sec    0    946 KBytes
[SUM]   9.00-10.00  sec   836 MBytes  7.00 Gbits/sec    0
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.44 GBytes  1.24 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.44 GBytes  1.24 Gbits/sec                  receiver
[  7]   0.00-10.00  sec  1.37 GBytes  1.18 Gbits/sec    0             sender
[  7]   0.00-10.00  sec  1.37 GBytes  1.17 Gbits/sec                  receiver
[  9]   0.00-10.00  sec  4.08 GBytes  3.50 Gbits/sec    0             sender                                  [  9]   0.00-10.00  sec  4.08 GBytes  3.50 Gbits/sec                  receiver
[ 11]   0.00-10.00  sec  1.29 GBytes  1.11 Gbits/sec    1             sender                                  [ 11]   0.00-10.00  sec  1.29 GBytes  1.11 Gbits/sec                  receiver
[SUM]   0.00-10.00  sec  8.18 GBytes  7.03 Gbits/sec    1             sender
[SUM]   0.00-10.00  sec  8.17 GBytes  7.02 Gbits/sec                  receiver
                                                       iperf Done.

Results

Test Result Retransmits
Single stream, Plex → Unraid, 10 s 7.03 Gbit/s (8.18 GB) 0
Single stream, Unraid → Plex (-R) 7.80 Gbit/s (9.09 GB) 1
Single stream, Unraid as client → Plex 7.83 Gbit/s (9.11 GB) 0
4 parallel streams, total 7.03 Gbit/s 1
Individual streams in the parallel test ~1.2, ~1.2, ~3.5, ~1.1 Gbit/s n/a

Speed guide

Reference point Speed
1GbE in practice ~0.94 Gbit/s (~110 MB/s)
10GbE theoretical TCP maximum (1500 MTU) ~9.4 Gbit/s (~1.17 GB/s)
USB 10GbE adapters, typical 6-8 Gbit/s
THIS TEST 7.0-7.8 Gbit/s (~875-975 MB/s)

Speed benefit

Unraid source/target Typical speed Benefit vs 1GbE
Single HDD in array (read) 150-250 MB/s ~1.5-2x
Array write with parity 40-100 MB/s None (parity limits)
SATA SSD cache ~500 MB/s ~4-5x
NVMe cache up to ~875 MB/s (link-limited) ~7-8x

Summary

  • Tdarr: pulling source files to the node and pushing finished files back runs at NVMe speed, and multiple workers no longer fight over one gigabit link. The transcode itself is still limited by the encoder.
  • Downloads and moves: files on the NVMe pool transfer to the Plex box or between systems at near link speed. The final move to the HDD array is still limited by the drives and parity.
  • Isolation: bulk traffic runs on its own cable and does not compete with Plex streaming, Pi-hole or Tailscale on the main LAN.
  • Plex extras: faster library scans, thumbnail and intro detection, and snappier seeking in very large files.

Where it does not help: live Plex transcoding speed is set by the GPU, not the network.

1 Like

I use this Cable Matters RTL8159 adapter:

https://www.amazon.com/dp/B0G1WJFR5V

However, it does need a USB 3.2 Gen 2x2 (20Gbps) port to achieve 10Gbps network throughput. If your USB-C port does not support dual 10Gbps lanes you will be capped at ~7Gbps throughput.

1 Like

Ah yes, I believe I’m capped at 7 Gbps, but good to know for the future!

If you’ve got a recent AMD motherboard with USB 4, these are thunderbolt compatible and have gobs of bandwidth available. I use a sonnettech thunderbolt 3 solo10g in my USB 4 port because not enough pcie slots and it works brilliantly, full speed.