Hello, I recently converted my TrueNAS from dedicated hardware to a VM (Converting baremetal FreeNAS to virtual) and am having some trouble with slow network speeds.
Using iperf3, I can get maybe 2gbps consistently to the TrueNAS VM from any number of other machines both physical and virtual (using Mellanox ConnectX-2 cards). The strangest (to me) part is that even VM → VM iperf on the same vSwitch is slow, around 2gbps. All VMs are using the VMXNET3 adapter type.
I’ve done some digging around and found some tunables to try, but nothing seems to be working. Does virtualized TrueNAS just have poor networking performance? I’ll probably post in their forums as well, but wanted to start here.
On baremetal I could hit bursts of about 8gbps on SMB transfers, which usually leveled out to between 3 and 4 gbps. On iperf I could nearly max out the full 10gbe link.
When virtualized, I was hitting the speeds mentioned in my first post, but in SMB transfers i’m only getting around 1.2gbps. Getting lower speeds on file transfers than iperf is something I obviously expect.
What’s CPU utilisation on the hosting hardware like? The virtual networking can be impacted by CPU shortages (in some cases even to the degree of causing packet loss/significant latency).
If it’s otherwise unused fair enough, but if it has other VM workloads, maybe something to check.
CPU utilization looks fine. Total utilization on the vSphere host is generally under 30%. As for the VM, it has 4 cores and maxes around 50% during my testing.
If I had to venture a guess, It could be an issue where VMware is passing the load between the CPUs which can really hurt performance.
I ran into a similar issue with some older Westmere hardware running FreeNAS in ESXi 6.7 with an intel x550 T2 passed through. I would get about half 10Gb speeds. Taking CPU 2 out, I got near full 10Gb speeds. Unfortunately for this situation, shortly after I decided to switch to bare metal setups which I have been running since. It would have been interesting to try to figure out how to make it work with 2 CPUs and v-switches but being my only NAS, I wanted to get it back up ASAP
When you setup the VM what hardware did you pass through? Did you allocate the memory or dedicate CPU cores to it?
Hm I was starting to wonder about that yesterday, but I don’t really know enough to form any educated guesses. The VM has 32GB allocated and reserved, as I’m passing the HBA directly to it. I can try reserving the CPU and see what happens.
Bear in mind that the networking at the hypervisor level has nothing to do with the number of cores allocated to the virtual machine.
Not saying that’s necessarily the case in your instance after your prior post - but its something to bear in mind.
i.e., i would suggest that it may or may not be a problem you’ll fix with VM settings. you may need to check into stats at the hypervisor level - NIC driver, network switch (is your physical switch able to handle the throughput for example?).
I had similar disappointments trying to virtualize TrueNAS locally under VMWare Workstation (passing in raw disk with VMXNET3 adapters). I think I got the best results below when I gave the VM 5 cores and setup jumbo frames (below).
I figured it was CPU speed limits on my older workstation, so was going to retry on a faster machine when I get the chance.
If I were doingi it in ESXi, however, I would try passing through the Mellanox ConnectX-2 to the VM, rather than using the virtualized networking – or if the NIC supported it, try out the PVRDMA adapter (although I don’t know how much that would do with TrueNAS):
% iperf -c truenas -P 1 -i 1 -f m -t 4 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to truenas, TCP port 5001
TCP window size: 0.06 MByte (default)
------------------------------------------------------------
[ 4] local 192.168.80.1 port 54790 connected with 192.168.80.129 port 5001
[ 5] local 192.168.80.1 port 5001 connected with 192.168.80.129 port 61281
[ ID] Interval Transfer Bandwidth
[ 5] 0.0- 1.0 sec 44.7 MBytes 375 Mbits/sec
[ 4] 0.0- 1.0 sec 41.2 MBytes 346 Mbits/sec
[ 4] 1.0- 2.0 sec 49.5 MBytes 415 Mbits/sec
[ 5] 1.0- 2.0 sec 23.8 MBytes 200 Mbits/sec
[ 4] 2.0- 3.0 sec 50.5 MBytes 424 Mbits/sec
[ 5] 2.0- 3.0 sec 28.5 MBytes 239 Mbits/sec
[ 4] 3.0- 4.0 sec 47.4 MBytes 397 Mbits/sec
[ 4] 0.0- 4.0 sec 189 MBytes 396 Mbits/sec
[ 5] 3.0- 4.0 sec 29.9 MBytes 251 Mbits/sec
[ 5] 0.0- 4.0 sec 128 MBytes 266 Mbits/sec
root@truenas[~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.80.129 port 5001 connected with 192.168.80.1 port 54790
------------------------------------------------------------
Client connecting to 192.168.80.1, TCP port 5001
TCP window size: 401 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.80.129 port 61281 connected with 192.168.80.1 port 5001
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 4.0 sec 189 MBytes 395 Mbits/sec
[ 6] 0.0- 4.0 sec 128 MBytes 267 Mbits/sec
root@truenas[~]# iperf -c 192.168.80.1 -P 1 -i 1 -f m -t 4 -d
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: -0.00 MByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.80.1, TCP port 5001
TCP window size: 0.16 MByte (default)
------------------------------------------------------------
[ 4] local 192.168.80.129 port 44259 connected with 192.168.80.1 port 5001
[ 5] local 192.168.80.129 port 5001 connected with 192.168.80.1 port 58098
[ ID] Interval Transfer Bandwidth
[ 4] 0.0- 1.0 sec 37.8 MBytes 317 Mbits/sec
[ 5] 0.0- 1.0 sec 33.6 MBytes 282 Mbits/sec
[ 4] 1.0- 2.0 sec 45.5 MBytes 382 Mbits/sec
[ 5] 1.0- 2.0 sec 38.7 MBytes 324 Mbits/sec
[ 4] 2.0- 3.0 sec 55.0 MBytes 461 Mbits/sec
[ 5] 2.0- 3.0 sec 37.5 MBytes 314 Mbits/sec
[ 4] 3.0- 4.0 sec 53.8 MBytes 451 Mbits/sec
[ 4] 0.0- 4.0 sec 192 MBytes 402 Mbits/sec
[ 5] 3.0- 4.0 sec 34.8 MBytes 292 Mbits/sec
[ 5] 0.0- 4.0 sec 145 MBytes 303 Mbits/sec
[SUM] 0.0- 4.0 sec 178 MBytes 373 Mbits/sec
So last night I decided to do what you mentioned @jeverett and just passthrough a NIC. But then i ran into a different problem. FreeNAS wouldn’t boot with the NIC passed through, and the installer wouldn’t even run. Both times I got this, then kernel panic:
mlx4_core0 unable to determine pci device chain minimum bw
I tried a different PCIe slot, no dice. I’m not sure what the issue is. It’s a dual CPU system with plenty of lanes. I even took a flash card out (unrelated to this) and put the NIC in its place. So, back to the drawing board? Sigh…
So I only messed with this once and it was a completely different set up, but I had to manually adjust the “rate” of the connection via cli… the MTP… I forget the variable I’ll have to look at home. I think I still had a issue with the VM in proxmox only allowing a certain speed for the connection as well so I had to adjust the MTU (maybe that’s right) via clu there as well. I never got to finish the project due to moving but that was the direction I was going… dunno if it helps at all.
Is that in FreeBSD? If so, maybe you’d have better luck with the Linux based TrueNAS SCALE. I seem to remember that Chelsio adapters were often recommended for FreeNAS, so maybe the BSD support for Mellanox is lacking (or vice-versa).
TrueNAS/FreeNAS are built on FreeBSD, so yes that error is being thrown in FreeBSD. This NIC worked fine on Free/TrueNAS on baremetal. It seems like the mellanox driver doesn’t like something about being a passthrough device.
Heh, wouldn’t be the first time Nvidia drivers sabotages the virtual environment. Maybe there is just some difference in initialization in the virtual environment. This thread seems to have the same error. They recommend some manual commands and settings – do any of these suggestion get any results?
I did see that post when I was searching around and unfortunately that doesn’t really help as I discovered that even the TrueNAS installer crashes with the NIC in passthrough. I’m chalking this one up to older hardware just not playing nicely. I posted on the TrueNAS forum as well and that was one of the opinions there.
I ordered an Intel X520 dual port card so hopefully I can pass that through and then be in business. I’ll update this post once that happens.
I finally got a chance to shut everything down and get the new Intel X520 card installed. I passed it through to the FreeNAS VM and it booted with absolutely no issues. The iperf test ran MUCH better:
Overall I’m very happy I went ahead and spent the money on the dual-port NIC. I was able to pass only one port through to the VM as well. So I might just use the 2nd port for the rest of the vNICs and take out the existing Mellanox card.