Advice Needed: Upgrading VFX Studio Network to 25Gb/100Gb!!

Hardware Hub Networking
1

Salutations to all dear members of LevelOneTechs, and greetings from Chile!

I’m here to kindly ask for your help in improving our office network and storage infrastructure.

Firstly, here’s the scoop: We are a small VFX studio in Santiago, Chile, working mostly on TV and movie VFX. Recently, we’ve added a few (four in total) workstations, and our 1 gigabit network is struggling with file transfers. We have decent 13900K machines with very fast NVMe local storage, but we usually work locally and only use the network to transfer files. This has become a bottleneck, often taking hours to copy our massive work files.

My boss has tasked me with the following:

  1. Upgrade our local network to 25Gb or 100Gb.
  2. Look into a NAS that can achieve performance close to a local NVMe drive.

Since we are a small team, we only need four ports, along with the necessary switch and supporting hardware. Eventually, we want to have a NAS for centralized storage with offline backup capabilities.

I have experience deploying a 10GbE network with TrueNAS and spinning drives but haven’t worked with NVMe-based NAS and the associated complexities.

For now, please help me with the following:

We are predominantly a Windows-based team, so plug-and-play or minimal setup would be greatly appreciated. I saw Wendell a few weeks ago discussing 25Gb and the various options available. Could you recommend some 25Gb or 100Gb NICs, an 8-port switch, and the related cabling (DAC cables or fiber+transceivers) that are known to work well together without issues? All our computers are in one room, a few meters apart, so we don’t need long-range solutions. It would also be great if I could buy this hardware from Amazon, as getting stuff shipped to Chile can sometimes be challenging.

I know Remote DMA is a thing, and I’m open to purchasing the license for “Windows for Workstations” if necessary. Please share your experiences on this matter.

What optimizations should I make (jumbo frames, etc.) to improve working with lots of small files? We typically work with EXR files that range from 2-200MB each, and usually, there are hundreds to thousands of these in a folder.

Thank you very much for your help and insights!

2

MikroTik is the hot ticket here and coincidentally used by your ISP’s for their backbone.

Genoa EPYC, sort through some of my posts to find the various configs I’ve built for just this. You’ll need all 128 PCIe Gen 5 lanes for what you are after.

Fiber SFP modules are your friend here, skip the copper for the NAS and multi gig.

You’ll end up saving an order of magnitude if you order from suppliers and have it shipped in. Amazon is not generally where you buy this tier of equipment.

3

Any NIC recommendations? I’m still unsure about the Mellanox vs. Intel debate, especially for 100Gb. Should I go with ConnectX-4 or Intel E810 for Windows? Ease of setup and configuration is a priority—less drama is better.

4

For Windows? Probably stick to Intel. Also the ConnectX 4 and below are dead to Nvidia now and won’t get more driver support, the Intel definitely will.

1 Like
5

Thanks so much for the info!

One last question—I know this can be a controversial topic, but what about DAC cables vs. fiber? Since all our systems are close together, can’t I use DAC cables instead of spending a lot on fiber transceivers and cables? I’m speaking out of ignorance here, so please let me know what you think.

this was what i was thinking about