Hey, long time forum lurker and fan of Wendell since TS here!
I’m building a server for personal stuff and can’t decide which OS I should use.
I should say that as a software developer I’m fairly tech savy but exclusivly on Windows, so all these Linux based systems are completely new for me.
I’m certain I will grow into it over time but don’t want to start with pages of scripts I don’t understand. Even if they are part of a tutorial - when things go south I wan’t to know what I’m doing.
Would be nice if some of you could give advice for my use cases (see below). Thanks in advance!
Hardware
CPU: Ryzen 7 3700X
RAM: 128GB 3600MHz
Board: ASUS Prime X570 Pro
GPU: GT 710 Silent
HDDs: 4x 4TB Seagate Ironwolf NAS
M.2 SSD: 1x 1TB Samsung 970 Evo Plus
10GbE: HP NC550SFP
Case: Fantec SRC-4240X07 (4HE)
Backup device: Synology NAS (2x 2TB Mirrored)
UPS: APC 1500VA
Use cases
Lightroom library (RAW photos and 4/8K video; stored on the server but edited on my workstation over 10GbE - this is very important to me as I don’t want to keep any data on my workstation)
storage for all my personal data
VMs for development (Windows 10 with Visual Studio, Eclipse/IntelliJ, Git, …)
1 VM for my wifes work (office stuff only)
and Dockers (PiHole, Steam cache, maybe Plex/Sonarr/Radarr, …)
12TB + 4TB parity is enough at this point
BUT it needs to be easily expandable in the future
My research so far
Upon reading a lot about the different systems Unraid seemed to be the way to go for me. So I tried it out to see how it feels and if it works with my hardware. And my first impression was VERY GOOD! Everything worked right out of the box - even headless (GPU will arrive tomorrow).
It’s very intuitive and easy to set up. So far so good…
BUT on further inspection I realized that the so called “cache drives” (which I need to saturate my 10GbE for smooth photo/video editing) are not the type of cache I thought it was. As a developer when I hear the term cache I think about a technical layer that holds a copy of the actual data in a faster type of memory and (asynchronously) writes every change through to the actual, slower type of memory. Normally as a user you can configure some parameters on how this cache behaves but never have to interfere with it’s basic functionality which is to make reads/writes faster for the front end.
Yeah and this is where Unraid thinks completely different and in my opinion misuses the term cache.
It’s basically another logical array of (ideally faster) drives which is not part of parity and whose data needs to be moved to/from the actual data array. This seems just seems extremely clunky and not very secure…
I know you could setup a backup job that copies the data from the cache drives to the array but that feels like a bandaid rather than an elegant solution. And for security I could get a second “cache drive” to mirror everything. That would cover failure of a cache drive but leaves the clunky mover / backup job on the table.
What I’m looking for
an OS that covers my use cases
and is overall not that hard to configure and maintain (you have to remember I’m an absolute Linux noob) -> Unraid seems to be ideal
a cache in the traditional sense which I don’t have to care about
every write goes directly to the cache and automatically gets (asynch) written through to the data storage
cache manages itself and retains data cached intelligently (often accessed = longer in cache)
maybe there could be an even faster cache layer in front of the M.2 cache by utilizing a set portion of the RAM (as I understand this is a Linux feature anyway)
What would you recommend? Any suggestions?
Or was I maybe too harsh with Unraids “cache”?
I thought Unraid copied its cache to the array each night around midnight? Not redundant, but at least flushes.
Did you try Freenas? The data arrays might be a bit slower, so it might not work for editing over Ethernet, even 10GB, but would be redundant?
Looking again at the services, I’m not sure it would run them as performantly as needed if using bhyve
Tiered storage is hard, as is figuring out what makes sense for your uses as a NAS.
I’d say if you can’t get behind what unraid does then Proxmox would be the next closest OS. Theres disadvantages to everything but I think unraid is a good compromise for the feature set it has.
For unraid I would recommend just putting the SSD in the array. The downside there is you have to manually tune what lives on it but this means the parity is never incomplete as it is when you run a cache.
Yeah thats my problem… I really like everything Unraid offers and it fits all my other needs perfectly. I simply don’t like the way caching works and furthermore don’t know how to apply their thinking to my use case.
Adding the M.2 to the array won’t work for me as I would be limited to this one cache drive’s capacity. My Lightroom library already has more than 1TB and will only grow bigger over time. And I only need quick access to the most recent (or most used files) not the whole lib.
And furthermore I may want to apply that logic to other shares too. So unless I make an all SSD array this wont do it for me.
From my limited knowledge I see these options (from most to least prefered)
An automated tiered cache like I’ve described (Unraid or other OS) btw thanks! “tiered” was the term I was looking for - english is not my native language >> Seems to be not achievable with Unraid
Any other ideas to dynamically cache the most recent / most used data >>Any ideas? (preferably with Unraid)
Split the Lightroom library into 2 seperate ones and manually move old footage from SSD to HDD. >>Inconvenient and may not be a solution for future caching use cases
Ideas anyone?
Edit: I’ll take a look at Proxmox in the mean time.
After several hours of research I think I’ll go with FreeNas & ZFS. L2ARC seems to be exactly what I’m looking for.
I know that Wendell has done a tutorial on ZFS with Unraid but as I’ve said I’m a Linux noob and really comfortable tinkering around at that level yet.
If you guys have further suggestions I’d be happy to hear them. But for now I’ll try FreeNas and see how it goes.
Freenas is now Truenas and soon Truenas Core a Linux version, I been using Freenas and moved to Truenas for about 2 years know. I have also have experience with unraid.
There is a lot going on with zfs so make sure you do your research on setting up your zpool and hardware.
https://www.youtube.com/playlist?list=PLjGQNuuUzvmug2-LMfh43ehP9nt8gmCSf
he’s got some really good info on Freenas/Truenas. It can get pretty intimidating, because its so flexible lots of tunables and ways to configure your zpool.
If you want good 10gb speeds you will need to get it setup right the first time around. My Truenas setup has 96GiB of ram, 40GiB connect-x 3 card, my zpool is striped mirrors of spinning rust and i can get 150MBs to 450MBs depends on the workload. My IOPs are great for VM’s and loading and working with photos Digikam. my local storage is minimal now since i work off the NAS theses days.
Hey ronclark, that sounds awesome.
Very helpful links. Thanks!
Yesterday I couldn’t fall asleep so I read a lot about FreeNAS and that it is now merged with TrueNAS. The BSD based version (named CORE) was released not long ago. (stable RC at least) Although I’m not fully aware of all the changes OpenZFS 2.0 seems to be a big deal.
But the most exciting thing was when I learned about TrueNAS SCALE. It is the Linux based version of TrueNAS that will come out later next year and will have much better VM support - thats exactly what I need!
And the best of it all is that we’ll be able to migrate from CORE to SCALE. So I can start out with CORE and later switch over to SCALE.
The decision is made guys. I’m definitely going with FreeNAS. It ticks every box for me!
A thing that I read about ZFS and L2ARC a while ago (not positive it still applies) is that L2ARC loses all its data on reboot, and it only holds data that has been ejected from ARC (which is RAM cache).
So one thing you may want to do for high performance is come up with something to run on boot which reads all of the files that you want to be fast, so that they are available from ARC or L2ARC later.
I think you are right, I’ve read that L2ARC gets cleared on reboot. But that’s not that important to me.
Ideally I won’t shut down/reboot very often
As you said I could write a script that reads my most used files. Maybe there is even way to read through L2ARCs content before reboot and read it on boot?
Thanks for that. I’ll look into this if it seems to get a problem. But for now I don’t think it will be.
I forgot to mention that L2ARC if I remember right the L2ARC takes a portion of your RAM and this why thy recommend maxing your ram out first and then if you need L2ARC add that.
So a new feature in Truenas 12 and openzfs is fusion pools for metadata storage on flash. My understanding is fusion pools will speed the lookup of Metadata. Not sure if that will help with your workload. @wendell said he’s going to make some videos on fusion pools.
“which I need to saturate my 10GbE for smooth photo/video editing”
What on earth are you editing? Unless you have a lab running multiple stations editing 4kraw or above video media you don’t need cache to saturate the 10gb for smooth editing.
