Is anyone else seeing problems like this in 22.12?
This is a weird one. In terms of change state I’m not sure when this stopped functioning properly, but I did two things recently:
- I made a bridge with no NICs attached
- I upgraded from SCALE RC1 to 22.12.0
I noticed this problem after the sum of those two things, but I’m not sure if either or neither of them have no bearing on my problem. I can see the following information from my switch,
The bond is functioning as expected and properly negotiating LACPPDUs
Channel group 11 neighbors
LACP port Admin Oper Port Port
Port Flags Priority Dev ID Age key Key Number State
Te1/1/1 FA 255 3cfd.fed7.bfcc 20s 0x0 0xF 0x1 0x3F
Te1/1/2 FA 255 3cfd.fed7.bfcc 16s 0x0 0xF 0x2 0x3F
It is reporting a 20 Gigabit connection:
#show interfaces port-channel 11
Port-channel11 is up, line protocol is up (connected)
Hardware is EtherChannel, address is 6c6c.d3c7.54b5 (bia 6c6c.d3c7.54b5)
Description: scale_prod_uplink
MTU 1500 bytes, BW 20000000 Kbit/sec, DLY 10 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, loopback not set
Keepalive set (10 sec)
Full-duplex, 10Gb/s, link type is auto, media type is N/A
input flow-control is on, output flow-control is unsupported
Members in this channel: Te1/1/1 Te1/1/2
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output 00:00:01, output hang never
Last clearing of "show interface" counters never
Input queue: 0/2000/0/0 (size/max/drops/flushes); Total output drops: 1986779
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 43000 bits/sec, 22 packets/sec
5 minute output rate 125000 bits/sec, 48 packets/sec
132187482412 packets input, 188018602975625 bytes, 0 no buffer
Received 11549916 broadcasts (7259457 multicasts)
0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog, 7259457 multicast, 0 pause input
0 input packets with dribble condition detected
119602936430 packets output, 168834842361229 bytes, 0 underruns
Output 16934790 broadcasts (0 multicasts)
0 output errors, 0 collisions, 5 interface resets
0 unknown protocol drops
0 babbles, 0 late collision, 0 deferred
0 lost carrier, 0 no carrier, 0 pause output
0 output buffer failures, 0 output buffers swapped out
On SCALE in the GUI I can see the bond reports 20Gb:
In the CLI, Ethrool reports 10 Gigabit as well:
root@prod[~]# ethtool enp68s0f0
Settings for enp68s0f0:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
ethtool enp68s0f1
Settings for enp68s0f1:
Supported ports: [ FIBRE ]
Supported link modes: 10000baseT/Full
Supported pause frame use: Symmetric Receive-only
Supports auto-negotiation: No
Supported FEC modes: Not reported
Advertised link modes: 10000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: No
Advertised FEC modes: Not reported
Speed: 10000Mb/s
Duplex: Full
Auto-negotiation: off
Port: Direct Attach Copper
PHYAD: 0
Transceiver: internal
Supports Wake-on: d
Wake-on: d
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes
But when I do an ip link show I am seeing a qlen of 1000 on the bond
root@prod[~]# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp68s0f0: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master b ond0 state UP mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
3: enp68s0f1: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq master b ond0 state UP mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff permaddr 3c:fd:fe:d7:bf:c d
4: usb0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN mode DEFAULT group default qlen 1000
link/ether 2e:4e:3f:19:e1:21 brd ff:ff:ff:ff:ff:ff
5: bond0: <BROADCAST,MULTICAST,MASTER,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
6: vlan60@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
7: vlan10@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether 3c:fd:fe:d7:bf:cc brd ff:ff:ff:ff:ff:ff
8: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode D EFAULT group default qlen 1000
link/ether ca:8b:a7:75:bc:cd brd ff:ff:ff:ff:ff:ff
9: macvtap0@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue sta te UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:22:12:da brd ff:ff:ff:ff:ff:ff
10: macvtap1@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:43:be:2a brd ff:ff:ff:ff:ff:ff
11: macvtap2@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:5e:d3:8f brd ff:ff:ff:ff:ff:ff
12: macvtap3@bond0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue sta te UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:1f:b2:c5 brd ff:ff:ff:ff:ff:ff
13: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 s tate UNKNOWN mode DEFAULT group default qlen 1000
link/ether fe:a0:98:01:94:59 brd ff:ff:ff:ff:ff:ff
14: macvtap4@vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:38:46:98 brd ff:ff:ff:ff:ff:ff
15: macvtap5@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:62:0d:b6 brd ff:ff:ff:ff:ff:ff
16: macvtap6@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:35:38:0f brd ff:ff:ff:ff:ff:ff
17: macvtap7@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:26:74:f9 brd ff:ff:ff:ff:ff:ff
18: macvtap8@vlan10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:43:54:3b brd ff:ff:ff:ff:ff:ff
19: vnet1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 s tate UNKNOWN mode DEFAULT group default qlen 1000
link/ether fe:a0:98:4a:1a:f7 brd ff:ff:ff:ff:ff:ff
20: macvtap9@vlan60: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue st ate UP mode DEFAULT group default qlen 500
link/ether 00:a0:98:13:0f:4f brd ff:ff:ff:ff:ff:ff
21: vnet2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 s tate UNKNOWN mode DEFAULT group default qlen 1000
link/ether fe:a0:98:6b:0a:70 brd ff:ff:ff:ff:ff:ff
When I attempt to do a file transfer from a computer, they seem to be limited to about 400mb, whereas before I was hitting much higher. confirming this, I ran an iperf test between the two:
C:\Users\nickf\Downloads\iperf-3.1.3-win64>iperf3 -c prod.fusco.me
Connecting to host prod.fusco.me, port 5201
[ 4] local 10.69.10.67 port 44812 connected to 10.69.40.8 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 47.9 MBytes 401 Mbits/sec
[ 4] 1.00-2.00 sec 47.5 MBytes 399 Mbits/sec
[ 4] 2.00-3.00 sec 48.0 MBytes 402 Mbits/sec
[ 4] 3.00-4.00 sec 47.8 MBytes 401 Mbits/sec
[ 4] 4.00-5.00 sec 47.2 MBytes 396 Mbits/sec
[ 4] 5.00-6.00 sec 48.0 MBytes 403 Mbits/sec
[ 4] 6.00-7.00 sec 47.1 MBytes 395 Mbits/sec
[ 4] 7.00-8.00 sec 47.9 MBytes 401 Mbits/sec
[ 4] 8.00-9.00 sec 47.8 MBytes 400 Mbits/sec
[ 4] 9.00-10.00 sec 47.5 MBytes 399 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 477 MBytes 400 Mbits/sec sender
[ 4] 0.00-10.00 sec 476 MBytes 400 Mbits/sec receiver
The TrueNAS system has an Intel X710-DA2 adapter and the Windows system has an X520-DA1. It was not performing this poorly before.