The goal is to use this NAS as our primary media storage in a video editing environment (primarily RED R3D files). We want to edit on the NAS i.e., not copy media to the local workstations.
How do we tell TrueNAS Scale to use the 25GbE NIC as a priority?
I have configured the NICs on the client workstation running Windows 10 Pro to prioritize the 25GbE NIC
Migrating data to the NAS, it will often use the 10GbE path instead of the 25GbE path. Throughput is what I would consider tpyical for 10GbE. However, even if I can successfully get the client and the NAS to use the 25GbE path, throughput is still at or below the 10GbE speeds ~ 500MB/s. I would expect >1GB/s. TrueNAS server showing 25GbE speed:
With 8 WD Gold drives in the NAS at RAIDZ1 I should easily get GB transfers - no? Max sustained on the WD Golds of 262MB/s. With one drive given up to parity in the NAS, that leaves 7 drives x 262MB/s = 1,834MB/s. Even if you only got 60% after parity, that’s still north of 1GB/s transfer rates.
Yes, it does. I was asking whether there’s any point in using multiple NICs for the same task - e.g. because they’re selected to separate VLANs.
What you should do if you want to only use the 25Gbit NIC for File Sharing:
Configure it with a Static IP in the Networking Section
(Remember to set Route and DNS if you were using DHCP)
Go to System Settings → Services → SMB → Edit → Advanced and bind it to that IP
In that case I’d also recommend that you go to System Settings → General and set the Web Interface Address to the one of your Broadcom NIC if you can put it on a mangement vlan, so you can separate Management and File Sharing.
About your Speed Issues:
Again, Go to System Settings → Services → SMB → Edit → Advanced
→ Under Auxiliary parameters, add server multi channel support = yes to be safe
What are the Settings on your Dataset(s)?
→ Atime and Deduplication need to be off on your pool and dataset
→ Compresion and Encryption should be fine with that beefy CPU, but it wouldn’t hurt to test
I’d write to a Dataset, or multiple if you want to test out different settings, using dd and /dev/random as a source.
Alternatively, pass through the Dataset Host Path to a Linux VM with the Phoronix Test Suite. But keep in mind that you need to test the passed through storage, VM Disk Performance will be misleading.
Edit: If you want to test SMB on the machine, create a bridge interface with the nic that you use for file sharing as a member and put a Windows VM on that bridge too.
That would rule out networking as a cause for slow SMB speeds.
I’ve seen rare occasions where I needed to try a second time, but generally setting up a bridge works fine for me.
It’s how I run my Dockers and VMs Networking right now.
What is a bit tricky is that you have to test the settings. So if you change your management IP, you have to switch over quickly enough to confirm the changes.
In any case, you could also configure this from the Shell via IPMI.
Should you need any extra guidance, feel free to ask.
What about just going to a single NIC - the Intel 25GbE?
Of course the in-bound IPMI would stay, but TrueNAS really doesn’t appear to care about that one. Seems there is a history of multiple NICs on the same subnet with TrueNAS Core.
I adjusted my reply above to reflect that, you’re right that multiple NICs on the same subnet can be a problem, I was assuming something about your setup.
If you want to also have a Bridge:
Put one of the 25Gbit NICs, or two of them as an LACP, as a Bridge Member
Give the Bridge a Static IP Alias for the Host System
