I am sure I saw something very close to this the other day on the forum but I cannot find it anymore.
I have tons of DVDs and blu rays and I would like to put those movies on my network as uncompressed MKVs, that is like 25-30GB per movie and shows are a lot worse of course. That is why I plan to split up my data into “important things I need fast access to at all times” and “stuff that is nice to have on the network but it’s not that bad if it explodes”.
Two separate systems:
I’m gonna stuff almost all of my harddrives into a big tower and run them JBOD or something like that with minimal data security and maximum capacity. So that system is covered. It will also be in a different room so I don’t have to listen to spinning rust all day.
Now for the fast and secure thing. This will be part of my rack right next to my desk.
The idea is to run only solid state storage for that. I am gonna build that system around my Asus Prime X370 Pro that just got replaced by the Taichi Ultimate. It will run a Ryzen 5 2600, some GPU, my LSI controller, a 10Gbe NIC and ECC memory (hopefully, have to try that).
I plan to buy four 2TB Crucial MX500 drives and run those in the ZFS equivalent of RAID5.
My main concern is:
Does ZFS do bad things to SSDs that would have no effect on HDDs?
But also:
Does anyone already have experience with a setup like this?
I think TRIM on zfs has not quite been released into latest zfs version.
Edit: the freebsd version does, zfsonlinux does not yet but there is a pull request
I think it is just ZOL that lacks trim, at the moment.
As long as you are easy on your drives, shouldn’t be a problem.
If you are turning them over a lot, it might be.
I would say for now, create smaller partitions on the drives, to allow the inbuilt wear levelling space to operate, and then expand the partitions later, when the trim support lands.
Hopefully before the next ice age
Dammit, I was trying to get away from BSD in general and FreeNAS is the only thing left.
But at least it is nice to know that it basically works and is probably getting there in the not too distant future.
I will build the system over the next week I hope. So far I just have two 500GB MX300 SSDs here and I will test with that first before sinking all the moneys into storage.
I also found the thread I was looking for with some help.
And it seems that TRIM is the last big thing that’s in the way.
… well, and NAND pricing obviously.
But it is close to where it was two years ago… yay.
8x WD Red 4TB HDDs for software and video storage in RAIDZ2. Video is mostly straight blu-ray rips, some DVDs and downloads when Blu-ray is not (yet) available.
2x Crucial MX200 250GB SSDs in mirrored mode for important data
4x Samsung 850PRO 256GB SSDs in RAIDZ for music. -> only added these last summer
Can’t say I’ve noticed any problems after almost 3 years of using the Crucial ones. The Samsungs seem to be holding up just fine too.
I installed the SSDs without worrying much. Then again I’m pretty comfortable risking some 250GB SSDs, not sure I’d be willing to gamble like that on 2TB ones without doing proper research first.
To me it wasn’t about raw transfer speed, after all my gigabit network is bottlenecking even the HDDs. But responsiveness was key. When I double-click a .pdf file or a song on the NAS, I want it to open instantly, not wait half a second for the read heads to get in position.
I did go with RAIDZ2 on the HDDs because I don’t want to re-rip my collection if anything goes wrong. Also IIRC RAIDZ helps prevent bitrot because it allows the filesystem to fix any issues using the parity data.
And yes, ZFS would be great for the HDD data dump but I don’t want to invest a lot into that box for memory. How do you get away with 16GB for that amount of storage? I thought 8GB for freeNAS plus 1GB per each usable TB of storage was the rule of thumb.
I had heard “1GB per TB of storage, 8GB minimum”.
I started out with 16GB (because I couldn’t get more from the same shop at that time) and wanted to upgrade to 32GB as soon as the modules would be back in stock, but I never seemed to run into any issues so I ended up sticking with just my 16GB.
Perhaps @wendell or other ZFS experts could shed some light on how important the 1GB per TB rule is and what risks I’m running by using only 16GB?
I haven’t noticed any issues since I built it, so I assume the lack of RAM will only affect performance (which is not an issue for my NAS seeing as it’s limited by my network’s speed)
Yes, less RAM means less space for ARC, and the ARC is important because ZFS fragments your files when you modify them and can’t defragment. In a situation where the data doesn’t change (e.g. media archiving) or where the storage could still be “fast enough” despite fragmentation (e.g. SSDs) you don’t necessarily need very much RAM for ZFS.
I did gather that you were going to run two systems from the original post, but this is much clearer indeed.
Ah, that explains it. So basically I’m just lucky that my use case allows me to get away with less RAM.
So do you reckon I should get another 16GB or not?
If you are happy with it as is then I don’t see a reason to.
A more reasoned, but quick and loose estimate: First, look up the size of your ARC. ZFS on Linux uses up to 50% of system memory by default, but I think FreeNAS uses a higher limit. ARC can only help you on the second (or subsequent) time you access the same data. So what is the total size of the files that you look at more than once over a short time period? If it is larger than your ARC, then adding more memory could help make your system faster.
This is a loose interpretation because ZFS can prefetch for sequential reads, ARC stores both data and metadata, and a “short” time period is relative (ZFS will try to balance between caching frequently and recently used data), but it still gives you a more specific recommendation than “1 GB per TB.”
I am running a RAIDZ across four EVO 960 500Gb on my workstation - works great but the performance degradation is there because of the lack of trim support. Hopefully that update will come soon!
