Help me choose a NAS/Server OS for 10GbE

Hey, long time forum lurker and fan of Wendell since TS here!

I’m building a server for personal stuff and can’t decide which OS I should use.

I should say that as a software developer I’m fairly tech savy but exclusivly on Windows, so all these Linux based systems are completely new for me.
I’m certain I will grow into it over time but don’t want to start with pages of scripts I don’t understand. Even if they are part of a tutorial - when things go south I wan’t to know what I’m doing.

Would be nice if some of you could give advice for my use cases (see below). Thanks in advance!


  • CPU: Ryzen 7 3700X
  • RAM: 128GB 3600MHz
  • Board: ASUS Prime X570 Pro
  • GPU: GT 710 Silent
  • HDDs: 4x 4TB Seagate Ironwolf NAS
  • M.2 SSD: 1x 1TB Samsung 970 Evo Plus
  • 10GbE: HP NC550SFP
  • Case: Fantec SRC-4240X07 (4HE)
  • Backup device: Synology NAS (2x 2TB Mirrored)
  • UPS: APC 1500VA

Use cases

  • Lightroom library (RAW photos and 4/8K video; stored on the server but edited on my workstation over 10GbE - this is very important to me as I don’t want to keep any data on my workstation)
  • storage for all my personal data
  • VMs for development (Windows 10 with Visual Studio, Eclipse/IntelliJ, Git, …)
  • 1 VM for my wifes work (office stuff only)
  • and Dockers (PiHole, Steam cache, maybe Plex/Sonarr/Radarr, …)
  • 12TB + 4TB parity is enough at this point
  • BUT it needs to be easily expandable in the future

My research so far

Upon reading a lot about the different systems Unraid seemed to be the way to go for me. So I tried it out to see how it feels and if it works with my hardware. And my first impression was VERY GOOD! Everything worked right out of the box - even headless (GPU will arrive tomorrow).
It’s very intuitive and easy to set up. So far so good…

BUT on further inspection I realized that the so called “cache drives” (which I need to saturate my 10GbE for smooth photo/video editing) are not the type of cache I thought it was. As a developer when I hear the term cache I think about a technical layer that holds a copy of the actual data in a faster type of memory and (asynchronously) writes every change through to the actual, slower type of memory. Normally as a user you can configure some parameters on how this cache behaves but never have to interfere with it’s basic functionality which is to make reads/writes faster for the front end.

Yeah and this is where Unraid thinks completely different and in my opinion misuses the term cache.
It’s basically another logical array of (ideally faster) drives which is not part of parity and whose data needs to be moved to/from the actual data array. This seems just seems extremely clunky and not very secure…

I know you could setup a backup job that copies the data from the cache drives to the array but that feels like a bandaid rather than an elegant solution. And for security I could get a second “cache drive” to mirror everything. That would cover failure of a cache drive but leaves the clunky mover / backup job on the table.

What I’m looking for

  • an OS that covers my use cases

  • and is overall not that hard to configure and maintain (you have to remember I’m an absolute Linux noob) -> Unraid seems to be ideal

  • a cache in the traditional sense which I don’t have to care about

    • every write goes directly to the cache and automatically gets (asynch) written through to the data storage
    • cache manages itself and retains data cached intelligently (often accessed = longer in cache)
    • maybe there could be an even faster cache layer in front of the M.2 cache by utilizing a set portion of the RAM (as I understand this is a Linux feature anyway)

What would you recommend? Any suggestions?
Or was I maybe too harsh with Unraids “cache”?

Your input is highly appreciated!

I thought Unraid copied its cache to the array each night around midnight? Not redundant, but at least flushes.

Did you try Freenas? The data arrays might be a bit slower, so it might not work for editing over Ethernet, even 10GB, but would be redundant?
Looking again at the services, I’m not sure it would run them as performantly as needed if using bhyve

1 Like

Unraid has a so called mover job that runs in specific intervals to either move data from cache to array or vice versa. (default = once a night)

Each share has to be configured in one of these ways:

  • Only = data in cache only; no parity so data lost on drive failure
  • Yes = writes to cache / mover moves from cache to array
  • Prefer = writes to cache / mover moves from array to cache
  • No = no caching

So there is basically no real intelligence involved as I would have liked.

1 Like

Tiered storage is hard, as is figuring out what makes sense for your uses as a NAS.

I’d say if you can’t get behind what unraid does then Proxmox would be the next closest OS. Theres disadvantages to everything but I think unraid is a good compromise for the feature set it has.

For unraid I would recommend just putting the SSD in the array. The downside there is you have to manually tune what lives on it but this means the parity is never incomplete as it is when you run a cache.

1 Like

Thanks for your input!

Yeah thats my problem… I really like everything Unraid offers and it fits all my other needs perfectly. I simply don’t like the way caching works and furthermore don’t know how to apply their thinking to my use case.

Adding the M.2 to the array won’t work for me as I would be limited to this one cache drive’s capacity. My Lightroom library already has more than 1TB and will only grow bigger over time. And I only need quick access to the most recent (or most used files) not the whole lib.

And furthermore I may want to apply that logic to other shares too. So unless I make an all SSD array this wont do it for me.

From my limited knowledge I see these options (from most to least prefered)

  1. An automated tiered cache like I’ve described (Unraid or other OS)
    btw thanks! “tiered” was the term I was looking for - english is not my native language :slight_smile:
    >> Seems to be not achievable with Unraid

  2. Any other ideas to dynamically cache the most recent / most used data
    >>Any ideas? (preferably with Unraid)

  3. Split the Lightroom library into 2 seperate ones and manually move old footage from SSD to HDD.
    >>Inconvenient and may not be a solution for future caching use cases

Ideas anyone?

Edit: I’ll take a look at Proxmox in the mean time.

You may want to look into low level caching solutions like lvmcache, bcache, or zfs

After several hours of research I think I’ll go with FreeNas & ZFS. L2ARC seems to be exactly what I’m looking for.

I know that Wendell has done a tutorial on ZFS with Unraid but as I’ve said I’m a Linux noob and really comfortable tinkering around at that level yet.

If you guys have further suggestions I’d be happy to hear them. But for now I’ll try FreeNas and see how it goes.

1 Like

Freenas is now Truenas and soon Truenas Core a Linux version, I been using Freenas and moved to Truenas for about 2 years know. I have also have experience with unraid.
There is a lot going on with zfs so make sure you do your research on setting up your zpool and hardware.
he’s got some really good info on Freenas/Truenas. It can get pretty intimidating, because its so flexible lots of tunables and ways to configure your zpool.
If you want good 10gb speeds you will need to get it setup right the first time around. My Truenas setup has 96GiB of ram, 40GiB connect-x 3 card, my zpool is striped mirrors of spinning rust and i can get 150MBs to 450MBs depends on the workload. My IOPs are great for VM’s and loading and working with photos Digikam. my local storage is minimal now since i work off the NAS theses days.

Here is a good thread on tweaking for 10gb networking performance

zfs cache info


Hey ronclark, that sounds awesome.
Very helpful links. Thanks!

Yesterday I couldn’t fall asleep so I read a lot about FreeNAS and that it is now merged with TrueNAS. The BSD based version (named CORE) was released not long ago. (stable RC at least) Although I’m not fully aware of all the changes OpenZFS 2.0 seems to be a big deal.

But the most exciting thing was when I learned about TrueNAS SCALE. It is the Linux based version of TrueNAS that will come out later next year and will have much better VM support - thats exactly what I need! :smiley:

And the best of it all is that we’ll be able to migrate from CORE to SCALE. So I can start out with CORE and later switch over to SCALE.

The decision is made guys. I’m definitely going with FreeNAS. It ticks every box for me!

Once again thanks for all your input!

A thing that I read about ZFS and L2ARC a while ago (not positive it still applies) is that L2ARC loses all its data on reboot, and it only holds data that has been ejected from ARC (which is RAM cache).

So one thing you may want to do for high performance is come up with something to run on boot which reads all of the files that you want to be fast, so that they are available from ARC or L2ARC later.

Pretty sure thats the case still.

Maybe its in our best interest to summon @freqlabs

I think you are right, I’ve read that L2ARC gets cleared on reboot. But that’s not that important to me.

  1. Ideally I won’t shut down/reboot very often

  2. As you said I could write a script that reads my most used files. Maybe there is even way to read through L2ARCs content before reboot and read it on boot?

Thanks for that. I’ll look into this if it seems to get a problem. But for now I don’t think it will be.

Support for not clearing on reboot has been merged, I think it is going to be a part of the upcoming 2.0 release of OpenZFS.

1 Like

I forgot to mention that L2ARC if I remember right the L2ARC takes a portion of your RAM and this why thy recommend maxing your ram out first and then if you need L2ARC add that.

Here is a good video on L2ARC from Puget Systems testing.

So a new feature in Truenas 12 and openzfs is fusion pools for metadata storage on flash. My understanding is fusion pools will speed the lookup of Metadata. Not sure if that will help with your workload. @wendell said he’s going to make some videos on fusion pools.

1 Like

Oh that’s nice!

So with 128GB RAM and 1TB persistent L2ARC I should be more then fine. That build is going to be overkill in any way :smiley:

Yep L2ARC is persistent in TrueNAS 12

A 1TB L2ARC is going to eat up a lot of RAM though, be sure to weigh the cost vs the benefit for your working set size.

“which I need to saturate my 10GbE for smooth photo/video editing”
What on earth are you editing? Unless you have a lab running multiple stations editing 4kraw or above video media you don’t need cache to saturate the 10gb for smooth editing.