Proxmox - Maximizing performance wtih 10G networking

I have a PVE 4 node cluster setup with a 3 node FreeNAS storage on 10G network, and I’m not happy with the backup performance. Iperf3 shows I’m getting 10G network speed between the PVE cluster and the FreeNAS boxes, and I can copy files to the NAS at around 500MB/s, but my backups run about 100-125MB/s. The PVE hosts have 3 networks, a 1G for corosync, 10G for the guests (the bridge interface), and a 10G for the SAN to the FreeNAS boxes. Iperf3 confirms that all three networks run at full speed.

There’s no chance that backup and migrate are being pushed through the sync network is there? The only reason I set that up in the first place is I was getting the occasional glitch, but since they changed out the sync technology with the new version maybe I don’t need it anymore?

What version of FreeNAS are you running? Ixsystems released a blogpost about 11.3 having materially better replication and copy algorithm and I recall something about 100Mbit caps. I may he talking crap or unrelated but worth checking.