Upgrading from 10Gbe to 40gbe or 100gbe for NFS, SMB, AFP

Hardware Hub Networking
, ,
1

Hello,

1st post here, but have been a fan of LevelOne for years now, thanks for all the the great information.

Apologies in advance if this is a bit long-winded, I’ll try to keep this as concise as possible.

The goal of this post is to gather any advice about the best route to choose when upgrading to a faster networking fabric, whether 40Gbe is the answer or if 50/100Gbe is the better move, and also how mature NFS and SMB are presently to accommodate those kinds of speeds.

Some Background:

Our video post production studio has about 12 clients, mostly macs and have one Linux CentOS 6.4 workstation running Baselight color grading software. Our bandwidth needs per client are fairly extreme, as all of the projects we work on are conformed back to camera original 4k, 6k, or 8k raw files (RED, Arri, Sony Venice, etc) or 10-bit DPX or EXR image sequences.

We built out our 10Gbe network about 5 years ago as an experiment, long before its massive adoption in our industry. Starting with netgear switches, connecting via cat6 to our shared storage, which has gone through a few different configurations of hardware. Originally we were serving from a quad core macMini with a TB2 to 10Gbe PromiseSan, connected to a 24-bay Highpoint rack.

This worked great but was bottlenecked to about 700MB/s due to the Highpoint chassis only having a PCI 2.0 x8 slot for the raid controller, combined with latency from the thunderbolt 2 pipeline etc. So about 3 years ago we migrated to our present shared storage box, the Qnap 1282T3 NAS, with the qnap dual-port 10Gbe NIC link aggregated to the same netgear (XS716T) switch. I was a little apprehensive to move to qnap as our main production storage, considering most studios in our industry have historically leaned on more expensive fiber SAN turnkey solutions such as Studio Network Solutions EVO, but the Qnap NAS 10Gbe solution has worked very well for the last few years. We are seeing speeds of around 900 MB/s read and write on average from all of our mac client boxes connected via AFP. (more on that later)

Current NAS:

tvs-1282t3

Within the 1282T3, it has 2 x4 pci 3.0 slots, and 1 x8 pci 3.0 slot. In the x8 we have the qnap (QM2-2P-384) nvme host card, with 2x 2TB EVO 970 nvme in RAID1, and assigned to r/w cache within QTS. The other x4 pcie slot has the qnap (LAN-10G2T-X550) 10Gbe NIC.

For the storage pool I have 8x 10TB Seagate Ironwolf drives in RAID6, along with 4x 2TB EVO 860 SSD’s setup in RAID5. I have qTier setup within QTS, using the SSD volume as the high-speed tier, and the spinners as the cold storage.

NFS/SMB/AFP:

We have experimented quite a bit from the mac side and a basic average of read speeds via 10Gbe:

AFP: 900MB/s

SMB: 800MB/s

NFS: 600MB/s

Our Baselight (linux) workstation has a software requirement to use NFS. Our Flame workstation (macpro 7,1) can connect via SMB, NFS, or AFP. Those are the ONLY two workstations we need to connect over 40gbe, the rest of our network can stay on 10gbe.

Proposed New NAS:

tds-16489u

This (unlike the 1282t3) has a x16 pci 3.0 slot I can use for our nvme cache, plus an x8 slot we can use for the 40gbe NIC. It has built in 10Gbe that will serve to our Netgear switches and the rest of our 10Gbe network. The 40Gbe NIC would direct-connect to the flame and baselight workstations.

For storage, I am planning all SSD (no spinners) and leaning toward the WD red 4tb drives.

Goal:

I would like to upgrade to a fabric to better leverage the speed of our nvme cache. I am hoping for around 3-4 GB/s throughput minimum.

Questions:

  1. 40gbe vs 50/100Gbe

Considering our setup are there any pro/cons to either? Is 40 Gbe past its prime and foolish to deploy at this point? Is there any difference between the two in regards to NFS optimization or SMB multichannel?

  1. Is there anything else I am overlooking here? Any other recommendations? I have considered building my own unraid or freenas server as well, but am leaning toward qnap to take advantage of the QTS tools to utilize the 1282t3 for backups, snapshots etc.

Any advice or comments are appreciated, thanks!

1 Like
2

I only have experience with 40gig / 100gig nics in the datacenter.

knowing what we had to go through to saturate 100Gbps, I’d be surprised if NFS or SMB would scale that high. (would probably go with ceph and distributed storage for that).

also, 40 Gbps is way cheaper right now, and once you deploy singlemode for 40gbps, and transform your software to be able to saturate it, 100gbps will be here. (some hyperscalers are looking at 200g/400g optics right now, I give it 3, maybe 5 years at most before it’s standard to have 200 or 400g optics in machines - at which point there might be a surplus of old pcie3.0/pcie4.0 100g nics (same as there’s today with 40G pcie2.0/3.0 nics)

3

Thanks, are you saying you don’t expect NFS or SMB to be able to saturate 40Gb/s or 100Gb/s or both?

For us, even if NFS or SMB could hit 30gb/s it would be a 3x to 4x speed increase compared to our current 10Gbe which would be close to acceptable for our current needs.

4

Theses HP 649281-B21 ConnectX-3 40GB cards are dirt cheap on Ebay, thy run around the $25 to $40 range, I run these in my homelab with a switch with 40G/10G/1G

Not sure if having a full end to end 40G network is worth it switches can get really costly, Maybe a 40G connected storage and 10G workstations might make the most since.

I can tell you getting 40GB throughput is tough, you need the compute power one both ends.
https://fasterdata.es.net/performance-testing/network-troubleshooting-tools/iperf/multi-stream-iperf3/

Here is a switch with 2 40G ports.

https://www.servethehome.com/mikrotik-crs354-48g-4s-2q-rm-review/

1 Like
5

Fully agree with @ronclark. 40GbE is mature and cost effective. Look at a switch with a combination of QSFP+ and SFP+ or 10GbE and you can scale out multiple workstations and servers that saturate single client to single host throughput.

100GbE fibre is really only needed for backhaul or hyperscale and for your use case is overkill.

6

I think you’d need to tweak it to hit 40. I think 100 is a pipe dream that’s easier to fix by not using smb at all and reimplementing everything it does from scratch… at least today

7

Now that i am more awake and read over your first post in more detail.
Are you working off the NAS or just need to transfer files for storage?

I am no network storage expert, I just have my home lab experience and what i have watched other put up, that being said I am not seeing the tds-16489u maxing out a 40G let alone a 100G connection.

Take a look at this LTT Video and you can see some of the problems you can run into trying to get fast storage.
https://www.youtube.com/watch?v=xWjOh0Ph8uM

8

I’ve seen that video a couple times already, interesting stuff as usual from LTT, but I don’t feel like it is relevant to what i am trying to do. Linus was experiencing nvme raid problem with 20x nvme drives. I won’t be doing that. the max nvme raid I will have is 4x, and it won’t be used for storage it will be used for cache.

In regards to whether the 16489I is capable of saturating 40gbe, it wouldn’t necessarily be the tds-16489U alone (or any other qnap model) that would theoretically be maxing out the the 40Gbe. It would specifically be the nvme pci controller card (QM2-4P-384) used for cache. A pci 3.0 x8 slot is capable of at least ~6GB/s, the 4x nvmes would saturate that, and that would saturate 40Gbe.

Unless my thinking is off any enterprise level qnap model with a two PCI 3.0 x8 slots (one for 40Gbe NIC, one for nvme cache) should be able to saturate 40gbe with the cache theoretically. Though I think even without the nvme cache a raid10 or raid6 of 16 SSD’s would easily hit those numbers as well.

By that logic my bottleneck would be specifically with TCPIP and smb/nfs, and not hardware.

9

nvme is x4 PCIe, what you need to fine out if the motherboard supports PCIe bifurcation otherwise you will be able to use 4 of your 8 lanes per card in each slot. there are expensive cards that dont need PCI bifurcations
Theoretical vs real world, real world is always slower.

Storage is complicated subject, it all comes down too use case. if your working off the NAS you really need IOP’s which makes it harder to hit max sequential read/write. if your just want storage with max sequential read/write normally IOP’s go down.
unless you build @wendell storage array that has mountains of spinning rust

10

That card has the PLX bridge chip - Does not require a motherboard w/PCIe bifurcation, but I am trying to do find the info on which of these qnap motherboards support it natively.