Change my mind... 10Gbe direct connections to NAS?

retheoff · November 11, 2020, 9:25pm

I literally just started my upgrading to 10Gb here. I originally wanted a couple straight connections because I’m only using 2 systems going to 1 with storage but in the end I don’t think it was much more to add a $250 Mikrotik 309 with 8 ports and I then had way more options later.

Also, I’m sure you realize you don’t get 40Gb speeds with that setup. Unless I’m just missing something? (you mentioned dizzying 40Gb speeds) But with 4x 10Gb nic’s on the NAS, one going to each workstation, you only get 1x 10Gb link to each. Right? Unless you are setting up 4x bonded ports on each workstation. And on a switched connection, you still only get the 10Gb between, one at a time. I’d put in a switch. The 309 was easy and I only had to adjust the couple of 1Gb links I put in. Plus I mixed fiber and copper, depending on the systems. My Mac had the 10Gb copper, the new Ryzen I had also had copper, but the storage server needed an add-on so I put in a SFP fiber to the switch since it was cheaper.

So someone above mentioned IOPS. I’m sure it totally depends on how your software does writes. I have 2 SSD arrays in a couple servers here. One has 6 disks (3 mirrors) and the other is 4 (2 mirrors). When using 4k writes, just doing some quick dd tests, there’s a big difference. That’s partly due to the disks I think too because of their internal caching. But on the 4-disk array I only get 4Mbps on a 400Mb file while it does sync on the writes. On the 6-disk, its 13Mbps. There’s no way 1Gb connection will be bothered by that. Now, if I do 1Mb block writes, that’s way different. Or, if I turn off sync on the test, I get speeds in the Gb’s per second. FYI, I use ZFS, which does weird caching things, so that’s a factor too. (personally, I would only do ZFS, but thats just me)

Honestly, if I had one of my clients asking me all this, I’d litterally bring in some decent, but not enterprisy, SSD’s in 6-disk (3 mirrors) ZFS in an older Xeon server and just test it out. I’d even test it on 1Gb ethernet and see what it does. Of course, for me though, I’d be happy to do that and if it doesn’t work I’d use the SSD’s elsewhere (and I have a spare older xeon).

I may be way off on this, but got a hunch even with a RAID10 of SSD’s you’ll be hitting some bottlenecks becuase of all the files. Hundreds of small files read in per project, and all those accesses from multiple systems, that’s going to go slower. And just guessing, but there’s probably way more read’s from those systems, but then a single large 300+ Mb file write every few minutes. If the software does small blocks and issues sync’s, its going to go slower than 1Gbps I bet. And that’s why I’d just get 6x cheap (not too cheap) 1TB SSD’s and see what it does. I used 4x WD Blues 1TB’s on my 4 disk and 6x Micron 9100’s on the 6-disk. ZFS seems pretty happy with them. For regular RAID only do RAID10 for IOPS. If you have a system with 16G memory you could spare for tests, load up Unraid or Proxmox or FreeNAS and test it out.

This is super interesting… you totally derailed my work here! LOL I’m partly iterested because years back I wanted to go into recording engineering, started schooling, then left, then ended up in PC’s and networking. But my son plays in Logic Pro a lot and I tinker with it once in a while. He’s always finding ways of using math to build music, he’s fascinated by it. And he just graduated with a BS in Math, yay! I bet he’d go nuts hearing what you do.

Anyway, I hope some of my info helps. I mainly work with with small biz networks and do development and have never needed 10G anwhere yet, except my home lab. But I have built a bit of ZFS servers for VM’s for clients. There’s much smarter people here with specifics on these bottlenecks and disk array performance.

brendanwoithe · November 11, 2020, 11:51pm

Oh some of your info has definitely helped. And got me thinking. I feel like we need to be a little more scientific in our approach to planning the file server specs / network.

Wondering if you (or someone else) could help me with finding some tools (software) that I could do some real-world measurements in order to try figure out what is going on in regards to read/writes during use.

I’m making a lot of inferences / making a bunch of assumptions - but I’ll try take you through those, then it would be amazing to figure out some tests, so once and for all we can best plan the network (and of course, specs for the file server)

Some learnings we do know, just without much exacting data.

DAW’s read and write data from three main sources.

Projects (and the audio files that belong to that project)
Database (library) of audio (and the database library itself) - and very occasionally video
Samples being read (only) for sample based instruments.

These are all quite different disk usages, and we structure things in specific ways - and this is pretty much an industry wide thing.

Part 1. Projects.

Projects usually end up on their own drive. The audio that is in a project folder will be imported there. Some software makes extra meta-data files for each audio file, and these are a number of orders of magnitude smaller than the audio files.

A project is a large single file, with its own cache file (which can be rebuilt if deleted) - and thats about it.
Every 5 mins or so the whole project file is re-written, and the old one becomes a backup. Backups are sequential. I keep at least 100 backups. (500 minutes)
The project file size varies greatly depending on what is in the session. Large mix sessions (we are doing some kids animation TV at the moment) are around 30-50MB, the cache is 9MB, the pool of audio files used by the project is around 5GB (for a 23min program) which is spread over approx 1000 separate files and the single video file is around 3GB (prores)

I do not know how much audio is read in by the software on load (into ram). Is there any way of figuring this out? What sorts of tools could I learn / grab to see whats going on? I’m thinking of putting one of the projects onto our old synology and using some sort of packet sniffer to see what network activity is occurring during project load / playback.

These project files (and their associated video/audio files) are the most import thing for us to put on the server. Different workstations need access to the same files - although only one workstation at a time will work on the same project file. (Does FreeNas have a way of “locking a file” thats in use?)

Database. This is our pool / library of sound effects that we draw from to create the soundscapes for each project. Its not as big as one might think. The libraries can be quite expensive - especially when licensing them across multiple workstations. Our library is kinda small/average for the industry. 400GB and 80000 items for sound fx, and for musical sounds there’s about 200000 at 600GB total. The DAW scans this library and makes its own (local) database directory - so the software is not usually scanning the drive. The only reads that occur are when you actually drag an audio file (or part of an audio file) into the project file to use.
Samples. Much of the musical instruments we use are either sample libraries (relying on very large sample sets) or software instruments that either synthesize the sounds or use a combination of samples + synthesis. These are NOT stored on the server - they are local to each machine. Different composers use different sets - and it means that a composer generally will only use her/his own machine. Our sound designers always use a little bit of music, but they use the same three or four main instruments that are local and loaded on each machine. The network does not see any traffic for these.
On my own machine I have 2 x 4TB SSD’s just for my sample instruments, and its sitting at around 6.2TB used right now. The use of SSD’s for this greatly speeds up workflow (as well as the number of instruments that can playback at once! In some large compositions, we can have 100’s of tracks. Some tracks can have 100’s or even 1000’s of voices going at once, especially for granular based sampling / large piano libraries with tonnes of microphone options)

We do not ever want to put samples on the file server. It just doesn’t make sense - and I cannot believe we’d be able to make it work fast enough without a tonne of investment. Also, the agreements of installing the software means they can’t be put on a file server anyway!

There is one exception to this. There are increasing numbers of small samplers being used for sound effects - things like footsteps and the sounds of clothes rustles etc. These are tiny in comparrison to say orchestral music libraries, but we CAN license them for use across workstations. I would be interested in looking at this in the future - just to save on administering each machine.

Now - to some learnings.

I have tested running the DAW with the projects on spinning drives (measured at 130MB/s reads with blackmagic speed test) and SSD’s measured at 480MB/s reads, a 2xSSD RAID 0 array with 740MB/s reads, and a 2300MB/s reads NMVE drive.
There is a massive difference in usability going from the spinning drive to the single SSD. I cannot say I experienced any difference when then going to the RAID 0 array or the very fast NMVE drive. Now, this doesn’t mean that with an extremely complex session (say a large film mix) we wouldn’t see usability improvements - but we don’t do full scale film mixes in our studios.
What does that look like? Project opening times are greatly reduced. Under half. More like 33%. When using, we don’t get a spinning beachball / unresponsive 5secs or so when the project file is being written on the SSD or above as compared to the spinning disk. Importing audio files off a faster SSD containing the sfx library is also greatly improved. Now, it might just be seconds here and there, but it really does matter for creative concentration. Waiting 10 seconds for a large file to be imported vs 1 or 2 can mean the difference between staying in the zone or going out for a coffee to reset.
For general playback of a session, I really cannot tell any difference between running off the spinning drive and running off the SSD.
We’ve also run sessions off the synology (which gives us 80-100MB/s throughput) and the feeling is similar to the spinning drive - although at times it feels like the system is just generally more sluggish. The gui can slow down / things get a bit jumpy etc.

We use 48k 24bit audio files here almost exclusively. Some 96k files, but unlike music recording, where higher sample rates are often used, sound design and composition with sample libs are mostly 48k/96k. 24bit is the standard for bit depth, and internally the machines all run a 32bit float mixing and processing architecture - with some plugins using oversampling and much higher internal frequencies. None of this impacts the possible network use. Just CPU use! (Interestingly enough, music composition CPU use is kinda like games use. Its not brilliantly scalable across cores because of the real-time nature of things. Sample buffers (how “real time” it really is ) greatly effects CPU usage. We all want lower buffers but it seriously stresses a single core. This same behaviour is seen in all DAWS. However, doing sound design only (not playing in any music via a midi controller) can (generally) cope with much higher sample buffers - meaning things start to use more cores much better. Getting a CPU tht is good across both can be a challenge. The new 5950X looks totally amazing for DAWS. 3950’s / Zen 2 TR’s never really took off because of some esoteric problems / strange performance issues. There’s some really interesting discussions on this over at SCAN Pro Audio who are industry kinds at benchmarking for audio software.

http://www.scanproaudio.info/2020/02/27/2020-q1-cpus-in-the-studio-overview/

http://www.dawbench.com/

So where to from here?

As I said, I’d love to do some tests, both on my local machines and across our current 1GbE network. Any software (OSX) we can use would be appreciated.

Onto more networking topology questions. I may well do this. Question. In order to cut down on the cabling required, if I do want to network my secondary machines - is there any massive disadvantage to putting a small 4 or 8 port 10GbE switch in my studio and just running the single 10GbE line back to the file server?

:). Yup, that much I understand. Directly attaching, each machine has its own 10GbE link. If I go through a switch I could bond, but I’d think the performance differences would be negligible and add to complexity. The switch WILL mean that everyone is sharing a 10GbE link though - so total throughput of the system to the server would be less - but I’m not entirely sure this would impact real world performance of the systems in any meaningful way.

Or maybe it would. In which case - having a small switch just in the two rooms that use more than one computer - and then each “room” going back to the server on its own link could prove a better bet. Maybe. I do not know enough / nor have I been able to figure this out through research yet.

Any way of testing this that anyone can think of?

Thanks all - I’m learning a bunch. Any info I can share about my industry for others I’m also very happy to share!

MazeFrame · November 12, 2020, 12:25am

When the server itself only has a single 10G NIC, then it makes no difference.

brendanwoithe · November 12, 2020, 2:38am

Oh I’m aware of that. The server build I’m looking at will have 2 x 10GbE and 2 x SFP+ 10Gb ports…

To clarify, I’ve quickly sketched up our network - showing how there are 2 separate networks. One is just for the file server (its currently GbE but the plan is for it to be 10GbE) and the other is for connection to the internet / connecting other network devices.

You can see how we use small switches in two of the studios - and that each network just has one cable running to each studio/room (equals 2 network cables to each room).

Can anyone see any disadvantages of working like this / using our current ethernet wiring (its all cat 6e) ?

retheoff · November 12, 2020, 4:47am

@brendanwoithe that’s very detailed and informative! Wow! And interesting! I’ll have to re-read this a couple times to take it all in I think. But I’ve got some thoughts to start with.

For one, I wonder if you’d be better off having NVMe (or just good SSD’s) in thunderbolt/usb-c cases for each system. You don’t need to have 1 project open on more than 1 computer anyway. You can eject them and swap them around easy too. I know you want to share on the network, but it sounds like a lot of that data can’t be shared anyway, or doesn’t make sense to (everyone has their own data sets or licensing issues). In fact, the things I’d think should be shared are the libraries and media that don’t get written to, or rarely written. They’re just read in when a project opens, right? Those are the things that should be on a network share, not the project files, IMO.

For NVMe on Thunderbolt, or rather USB3 gen2 with USB-c connector, I’ve only tried one enclosure. I just ran a quick test with Blackmagic, formatted APFS (encrypted!), and I get 700-800MB/s writes and 700MB/s reads, which I think is roughly 5Gbps speeds (right?). Anyway, its fast, and thats with encryption too. It really depends on your enclosure though! Some are not good. This test has a 970 EvoPlus 1TB, and I’ll have to lookup the enclosure (glotrend?).

Here’s dd and blackmagic:
(4G file, 4k blocks writes)

dd if=/dev/zero of=test.perf bs=4096 count=1000000 conv=sync
1000000+0 records in
1000000+0 records out
4096000000 bytes transferred in 4.983765 secs (821868626 bytes/sec)

dd if=/dev/zero of=test.perf bs=4096 count=1000000
1000000+0 records in
1000000+0 records out
4096000000 bytes transferred in 4.894382 secs (836877873 bytes/sec)

Blackmagic is roughty the same:

Anyway, I bring that up because sharing on a network doesn’t ever seem to match up to performance tests. There’s network protocols involved before you hit the disks. And also, you did mention losing a few seconds is important to the creative process. If it is, maybe the NVMe enclosure route is worth investigating. I know, its not as convenient.

So let me just share a few things I’m finding on my new 10Gb network here. Keep in mind I havne’t done any tuning yet, and I have a whole lot of tech that may cause slowdowns (VM’s, ZFS, unoptimized sharing). I just plugged things in and went back to work, and I haven’t even put my shared data on it.

Here I have my Mac Mini 10G connected using NFS to my new VM server. This test is on NFS to a Linux VM, which has a raw disk image running on the underlying Linux host using ZFS. That ZFS pool is 2 mirrors of 2x WD Blue 1TB’s (pretty much RAID10).

Here’s a dd write test with/without sync. (Ran on Mac, over NFS, on a VM running on ZFS, HAHAH!)

dd if=/dev/zero of=test.perf bs=4096 count=100000 conv=sync
100000+0 records in
100000+0 records out
409600000 bytes transferred in 1.870827 secs (218940612 bytes/sec)

dd if=/dev/zero of=test.perf bs=4096 count=100000
100000+0 records in
100000+0 records out
409600000 bytes transferred in 0.836980 secs (489378558 bytes/sec)

Here’s the same using Blackmagic 5G file test (but that doesn’t test sync)

Watching the zpool iostat’s during part of that looks like this: (just one tiny snapshot, it changes greatly!)

                                                    capacity     operations     bandwidth 
pool                                              alloc   free   read  write   read  write
------------------------------------------------  -----  -----  -----  -----  -----  -----
korakk                                             710G  1.12T      0  6.71K      0   794M
  mirror                                           355G   573G      0  3.39K      0   406M
    ata-WDC_WDS100T2B0A-00SM50_20325B800799           -      -      0  1.73K      0   207M
    ata-WDC_WDS100T2B0A-00SM50_20325B804304           -      -      0  1.66K      0   199M
  mirror                                           355G   573G      0  3.31K      0   388M
    ata-WDC_WDS100T2B0A-00SM50_20325B804345           -      -      0  1.65K      0   194M
    ata-WDC_WDS100T2B0A-00SM50_20325B804359           -      -      0  1.66K      0   194M

When I run a simple iperf test between the two system’s I get this:

iperf3 -c magmoor 
Connecting to host magmoor, port 5201
[  7] local 10.9.8.10 port 54229 connected to 10.9.8.146 port 5201
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-1.00   sec   433 MBytes  3.64 Gbits/sec                  
[  7]   1.00-2.00   sec   426 MBytes  3.57 Gbits/sec                  
[  7]   2.00-3.00   sec   429 MBytes  3.60 Gbits/sec                  
[  7]   3.00-4.00   sec   425 MBytes  3.57 Gbits/sec                  
[  7]   4.00-5.00   sec   429 MBytes  3.60 Gbits/sec                  
[  7]   5.00-6.00   sec   423 MBytes  3.54 Gbits/sec                  
[  7]   6.00-7.00   sec   431 MBytes  3.61 Gbits/sec                  
[  7]   7.00-8.00   sec   424 MBytes  3.56 Gbits/sec                  
[  7]   8.00-9.00   sec   432 MBytes  3.62 Gbits/sec                  
[  7]   9.00-10.00  sec   419 MBytes  3.52 Gbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  7]   0.00-10.00  sec  4.17 GBytes  3.58 Gbits/sec                  sender
[  7]   0.00-10.00  sec  4.17 GBytes  3.58 Gbits/sec                  receiver

I am totally not matching up to Linus’s videos on his 10G stuff. But again, not doing any tuning here yet. I use CentOS 8 directly too, no Unraid or FreeNAS like Linus. I may even have PCIe lanes messed up becasue I have two GPU’s in there! OH, and I haven’t even messed with jumbo frames yet!

How to test IOPS, I don’t know. I’ve seen benchmarks online, but never realy needed to test any of that myself.

I keep looking over your use of the different files involved. I can’t get around this one thing: “the pool of audio files used by the project is around 5GB (for a 23min program) which is spread over approx 1000 separate files”

There’s a lot of overhead in openning a large amount of files. I wish I could say use one test or another, but you’ll never really know until you put your software on it and test it. Unfortunately I haven’t moved all my data onto the 10Gb network shares yet. I feel like its about 3-5x faster, but not 10x faster, which I more or less expect. I would like to put my development data and code on it and see how it “feels” while I work. But even the 10Gb ethernet speeds and backend disks won’t match the internal PCIe disks in the Mac, so… yeah, probably won’t work direclty on the network shares.

BTW. If you use SMB to your NAS, you may be able to see all the file locks and open file status using “smbstatus -v”. I use that on linux hosts to see all the open shared files quickly.

(this was a reply to your earlier post, just saw you posted another one)

retheoff · November 12, 2020, 5:00am

Oh FYI, on the overhead of large amounts of files… I thought I’d drag over a dev project to my share I tested earlier. Its got about 8,000 files and 2.5Gb in size. I’m 10 minutes in and its still says “estimating time remaining…”

brendanwoithe · November 12, 2020, 5:09am

So - interesting you mention this.
Since working with our current Synology has proven not great, we have created drives for different projects, and utilising blackmagic docks (4x2.5" to USB-C)

However, this is a nightmare and just WAITING for a large problem.
For something like a TV series, there are 26 episodes. While EP1 Edit might be being worked on in room 1, Ep2 Mix might be on in room 2 and Ep3 Broadcaster WIP might be worked on in room 3. So does that mean we need 26 different drives? Copying things from one drive to another is a great way to accidentally write over information when things are sync’d in the wrong direction.
We also have 200+ short turn around projects run every year (average 4 a week finished) and they bump between rooms as well. At any time a room might need to refer to older versions / or there might be multiple rooms working on different projects.

I’d love it to be just drives. It just isn’t feasible. Especially with complexity increasing, and us potentially using the 4th station as a workstation rather than a producers chair…

In the industry its known as sneaker net. I know only of one facility to pull it off for about 2 years back in 2016/2017 - using small thunderbolt SSD’s. They had 6 or 7 rooms and 40+ drives. When one project accidentally got wiped then the backup got wiped and some REAL project data was lost due to the way they had backups implemented, they invested $60k+ in a server/infrastructure, as their loss from the project was huge. This was a picture edit co in London, and pic edit is much harder on a network than sound by perhaps an order of magnitude. They went 40GbE fibre…)

More soon - I’m in the middle of a session

retheoff · November 12, 2020, 5:48am

Oh trust me… I get the need for a shared server. And I don’t know your workflow, but that does sound like a mess waiting to happen. I had the impression the seconds mattered more so I wanted to compare, knowing that it probably was a no-go. Someone with more 10Gb experience would probably need to chime in here. And at that, someone with audio and/or video editing on a NAS/server too. I know Linus Tech Tips had some videos on their 10Gb and video editing workstations. Maybe there’s more info on their forums too.

Anyway, hopefully those speed tests to my SSD pool over 10Gb helps get you an idea.

brendanwoithe · November 12, 2020, 6:23am

They most certainly do - thanks.

Audio shares many of the needs of video, but generally just larger numbers of much smaller files (but even then, 1000 files in 20 odd mins is only 50 files a minute, 1 per second… and I’m pretty sure the software is only reading the part of the file it actually needs… and I’m also fairly certain a decent amount is coming out of RAM / is read on project open. This is where I’d love some sort of tool that I can read the disk I/O’s by application in real time… so I can see whats actually going on. I’m more than certain that a 10GbE network will handle what I need it to, since 1GbE is 50% of the way there (and kinda useable ish but not really) - but figuring out the CPU and RAM requirements for the server is important. I think a high-ish end ATOM with tonnes of RAM might well be sufficient for FreeNas/TrueNas… I’m still learning.

deadvoid · November 12, 2020, 6:50am

I did a direct 4x 10GBase-T server to workstations with Cat6A years ago for a (visual) post facility, quite similar to yours, not using a switch is fine given they’re still in business until August this year. But since we used Flame & Baselight I didn’t design the server as online pool to work from.
Since you’re planning to work directly from the server, you might want to consider SFP+ for the workstations that don’t have built-in 10GBE, less power & better latency. Drawback is you need another NIC and PCIe slot in the server, and the SFP+ NICs are more costly.
A tip for macs, last time I checked (mid last year) Smalltree still sells Intel 10GBE NICs as well as updating driver for Mac.

I’m not at all familiar with DAWs though, in both hardwares and softwares, so for

I do not know how much audio is read in by the software on load (into ram). Is there any way of figuring this out? What sorts of tools could I learn / grab to see whats going on? I’m thinking of putting one of the projects onto our old synology and using some sort of packet sniffer to see what network activity is occurring during project load / playback.

I’m only guessing here: I don’t know if sound cards has memory in it, or if they’re utilised like GPU in video softwares, if they do then they’re using sound cards to process audio including keeping most of the tracks to stream to workstation’s RAM buffer, but parts of file(s), not loading all of them at once, since I suspect 5-10 minutes uncompressed tracks eat RAM very quickly.

I’ve an idea when I read your post minutes ago but I’ve ran out brain power now, so I’m stopping here.

Log · November 12, 2020, 7:14am

Even the old C2000 series atoms could handle it just fine (they were initially popular in the community, then it turned out there was a hardware bug that kills them). Anything newer is fine.

My first custom NAS with Debian and ZFS was based on the Asrock QC5000-ITX/PH which has an ancient AMD FT3 Kabini A4-5000 and had 8G of Ram, I was limited by my pool layout far before anything else. I’m pretty sure I’ve used a 10G sfp+ NIC in it before before I retired it because it only has 4 sata slots. It only has one pcie slot, so I’d have to choose between an HBA and 1G, or 4 drives and 10G.

The correct solution it to get as many pcie slots as you can, even if ITX boards are strangely attractive for some reason.

Note, the workload here is running and OS with ZFS storage with lz4 compression and occasional send/receives. Actual workloads are a whole different ballgame.

retheoff · November 12, 2020, 9:12am

Is that what the deal was?!!! I don’t remember the atom model, but those Supermicro 8-core Atoms from about 5 years ago, they were awesome! Except that I put in about 10 mini-server , my partner did even more, and about half of all of them failed, most after their warranty. Probably the worst boards I ever used. Stupid thing is, for the other half that kept chugging away, they’ve been some of the best boards I’ve used. I still use one for ZFS snapshot backups here. I got them because they were the same boards IX Systems was using on their little NAS servers.

If I were to do any file server like those again, I’d rather put in a low-end Xeon. 32G ram. It would cost a little more, but you’d have more PCIe slots. If you’re investing in 10Gb network, what’s an extra $300? If I had growing storage (who doesn’t!?), I’d want to be able to add another 6+ disk ZFS pool. The ITX Atoms have one slot, those old ones did.

I doubt you’d need 32G RAM, but about 3 years ago I told my partner, “never build servers with less than 32G anymore!” At the time we kept running into situations where 16G wasn’t quite enough, and the cost just isn’t enough to care about.

MazeFrame · November 12, 2020, 12:05pm

Very helpful. I have not understood this thread to represent that network setup.

Not that I could see. Then again, I am no expert.

retheoff · November 14, 2020, 8:54pm

@brendanwoithe , I just wanted to note that I solved my performance on 10Gb. I was wondering why I only saw 3Gb in iperf3. I was actually having 2 problems, but the main speed issue was due to TCP Window scaling that was disabled on my servers. Once I enabled that I got 9.x Gb/s in iperf3 and all over-network disk shares showed more expected speeds. I tested quick to single NVMe and an SSD on a share, and on the NVMe I got around 700MB/s read and write on the network. On the SSD it was about 450-500 MB/s. But the very cool thing for me was setting up my pool of old disks on ZFS. I have that pool setup with 5x 2TB WD’s and also 3x 4TB’s, each in raidz1. (so, 2 raidz1 VDEV’s in ZFS terms). That get’s me very usable performance for spinning drives on a network share.

Blackmagic tests over SMB share to VM with ZFS.
Non-encrypted:
550MB/s writes, 890MB/s reads.
Encrypted (I use that mostly here):
475MB/s writes , 650MB/s reads.

I’m finally happy with the 10Gb speeds! Also, this is not a good comparison for you, but I ran a couple tests with iMovie library over the network and it ran like I was on the local drive. It wasn’t opening 1000 files or anything, but it was very noticeably slow on 1Gb network before, so I wanted to see how it worked.

I know your setup will be different but thought I’d share anyway. Also, my setup is with 3 systems shared on a single switch. For what its worth, if I were setting up a network from scratch, I’d always opt to use a single switch if possible.

Report back when you get things running. I’m curious how it works out for you.

dwm · November 30, 2020, 3:46am

Nothing inherently wrong with multiple 10G connections. Though you’ll want to pay attention to PCIe lane and slot allocation for whatever you intend to use on the server side.

One issue that comes to mind: random I/O tends to be poor on consumer SSDs. I’ve no idea how much random I/O is happening in your workflow. Also no idea how much benefit you’ll see from cache (ARC in ZFS, for example) since it depends on the usage patterns of the software in question.

Question 1: what is your backup scheme? Let’s say there was a fire in the facility… do you lose everything?
Question 2: how might a revision control system (git, subversion, et. al.) aid you?

Question 1 is sort of obvious… at some point in a growth path, you want to start thinking like a larger enterprise when planning IT stuff. I can’t tell you what you need for the next few years, nor tell you how to spend your money, of course!
But… if your NAS went belly-up, can you recover? How quickly? If an individual Mac goes belly-up… where are its TimeMachine or CarbonCopy Cloner or Super Duper backups?

Question 2 might be less obvious, and I don’t work in audio/video but…

It’s really difficult to beat local NVMe for speed. Since time is money, to the extent possible I’m always aiming for having the assets I’m currently working on local to my workstation, with the previous revision(s) in central storage. Yes, my changes need to go back to a centralized location (for sharing, backups, etc.). And I gather you’re already doing some of that. But there can be many upsides to the I/O-intensive stuff happening locally (say to a 4x4 NVMe card), then synced back to the server manually (checked in to a revision control system), or automatically on a periodic basis (via cron or otherwise). Manual check-ins give you control over when something gets shared (so you aren’t sharing half-baked stuff), and also control over when the network is hit hard. Yes, it requires some discipline. But those of us who’ve been software developers for a long time do almost nothing without revision control. If the repository is off-site, all the better (I can recover from a local disaster like a fire, and I can also push/pull when I’m not in the building, say working from my Macbook Pro at a client site). It also means I can do maintenance work on the centralized storage without halting the work being done on the workstations; the workstations have everything they need for currently active work.

I ask these questions because there’s always more than one way to get what you need. And I always think of backups (and off-site backups) as a primary need. And I like having revision control, on almost everything, since it allows me to fork (branch), tag (labelled revision), have everything local (my ‘working copy’), control when I share with others (don’t give them access to stuff that’s not done that might waste their time), back out to previous revisions… all with explicit control. And sometimes it turns out that having a solid backup strategy can go a long way toward making the best use of the network, local storage and centralized storage. If all machines are backed up nightly to your server, for example, you know that hauling that server out of the building in a disaster (say an impending flood or wildfire) is sufficient for a full recovery other than the work that was done that day. And you can push backups from that server into off-site cold storage or similar at whatever frequency make sense and is within your budget. Or even pull hot-swap drives and sneakernet them off-site for long-term storage.

I don’t work in audio/video, but I do work on large software projects, and some medium sized ones that are my own. Most of them aren’t gargantuan in and of themselves, though some have large datasets checked in to revision control (git or subversion). When I need them, I pull the revision I want (usually the latest) from the repository. It’s speedy over 10G ethernet as long as the target is speedy (NVMe in my laptop or workstation). I then use it locally, which is very speedy (and impacts no one but me). If I modify it, I check it back in to the repository when it’s ready (often many times in a single day). For datasets that are smaller and/or don’t need revision control, especially read-only stuff, I’ll just keep them on the local server and access them with NFS or SMB.

I have both TimeMachine and Super Duper backups for the Macs (TimeMachine turned down to once every 2 hours, Super Duper nightly). For the linux and FreeBSD machines, level 2 incremental backups over the network every night, level 1 backups each week, level 0 backups every 4 weeks (all automatic). No Windows on bare metal, only Windows VM and those snapshots get saved as part of the host OS backups. All backups (including TimeMachine) going to one ZFS pool, except the backup of the main ZFS host (which goes to another ZFS pool on a separate host). One of the nice things about the linux and FreeBSD backups is that it’s sequential writes; it’s fast even on spinning rust. Backups of irreplaceable stuff (repository, personal photos, etc.) gets pushed to cold storage in the cloud (compressed and encrypted). Repository dataset gets ZFS snapshots which are sent to second ZFS server in-house too. I consider the in-house backups and ZFS snapshot send/receive my primary line of defense; they allow me to quickly recover from typical problems (workstation problem or replacement, for example). Cloud storage is to handle a site-wide disaster. And the automated network hammering (backups) all happens when no one is working, which tends to yield good coherency on stuff that matters (work) and is no-impact most of the time (unless someone is working at 4AM).

Anyway, just food for thought, sorry for the long ramble!