Compression is a property of a dataset (mountpoint, directory, filesystem are other terms for it) and gets inherited by all children (sub-directories if you so wish) by default but can be changed there too.
Compression is great because the data in the cache is stored in a compressed state, meaning you get more space out of your memory and cache device. I had 230GB of data stored in my 96GB ARC at some point just to give you an idea. But when talking VMs, Containers or my steam library, compression is just awesome.
I think TrueNAS ships with LZ4 by default, which is super fast and has an early abort if the data isn’t compressable. It’s like GBs per core per second. It’s basically free real estate. ZSTD is the new kid on the block with very good performance and better compression than LZ4, but performs worse for uncompressable data.
Every vdev has to rpovide their own redundancy. Your RAIDZ can handle one failed drive. But if you only have one drive as a special metadata+small block device and this drive fails, the pool fails because ZFS doesn’t have metadata anymore to ask the HDDs where everything is (generalizing).
Each vdev has to provide their own redundancy. If you want one drive of tolerance, you need a mirror (2 drives) or RAIDZ (3+ drives) for your special vdev.
Those drives actually look really good, their MTBF is higher and they have more cache (I can only judge from the datasheet though). That will also make it cheaper to expand it once vdev expansion is there and 2x8TB spare will make data transfer way easier.
I’m using Toshiba MG08 myself and got no problems in 9 months of basically 24/7 operation (3 different batches of drives). They are a bit noisy though, I wouldn’t want them doing random writes in my living room. I heard the Exos aren’t like that, but noise is always a bit subjective.
I keep spare HDDs as a backup pool (RAID0) in TrueNAS that gets plugged in once a week for replication/backup job. Always good to have backup. And ZFS backup is super fast because it only needs to transfer what has changed. TrueNAS GUI is really good in doing this with like 4 clicks or automate it all together.
The question would be how much space would I need for a special metadata device? Is a 100GB enough? It probably depends on the number of files too, but it’s kinda hard to grasp for me.
We’re talking about sub-100GB worth of metadata for a 32TB pool. And considering you mostly store media files (dataset with >128k record size, I use 1MB record size for my media), a special vdev just for metadata isn’t worth it.
People here use these special devices mostly because you can optionally also store small files there. And this is a very nice thing indeed if you know how bad HDDs perform when you tell them to get you 1.000 small files. Small files on fast SSD and the big chunks on HDDs is a nice thing to avoid the worst aspects of HDDs.
So if you feel like your HDDs are doing a lot of random reads and writes on small files, this option becomes very attractive.
I’d stick to those 32GB of memory and the HDDs for now and add a L2ARC or special vdev for performance tuning later. Or get a 500GB-1TB NVMe SSD and use it as cache, you can always remove it and repurpose for other things.
Yeah kinda hard to decide. In size it’s mostly media files, but I do have a substantial amount of small files in count too. Which will probably expand as I might migrate some stuff over from my server, so that my server stores less files (which it currently does store, without any redundancy). But I think your suggestion is valid and I should start with a 500GB (cache and boot) drive or so and expand from there.
To add some data: Media dataset: 6TB, 2k files; home dataset: ~150k files and expanding rapidly, 1TB (although the size can probably be reduced if removing the photos, etc from it)
Let’s say you store 30TB of media in your pool/movies datasets with 1M recordsize. That’s <5GB of metadata which is trivial and more of a foot note for 32GB memory. But if you got a 30TB database running on a 4k dataset, that’s 256 times more metadata, so you better get an EPYC board to accommodate your new TBs of metadata in memory. And we’re not talking about caching any “real” data yet
These are extreme examples, but they can give you an impression on the subject.
We all do. The benefit of small files is that they’re small. You can store a lot of (compressed) small files in 500GB, be it L2ARC or special vdev.
Motherboard:
I’d would strongly avoid ASRock on Intel as they’re multiple times made very strange decisions about hardware design on several products for the Intel platform.
I would however suggest something that uses Intel NIC to make your life much easier, MSI PRO Z690-A is a decent “normal specced” motherboard which could be a decent replacement.
SSD: Decent choice but I’d say you pay a premium, Crucial P5 Plus 1TB goes on sale quite frequently however also worth keeping in mind that either runs fairly hot you make sure you have decent airflow.
HDDs: Works great I have a pair myself
Case: Just grab something from Fracal Design’s Define-series, you’ll thank me later =)
PSU: I’d suggest that you have a look at Corsair’s RM(X) → RM???x series or Seasonic’s Focus Plus series. Both are very stable and quiet platforms backed with very generous warranty (10 years)
Intel is a much better choice than Ryzen in this case especially because of the hardware encoding.
Yeah I’d rather not go with ASRock too normally, but the CPU was low end enough that VRM shouldn’t be a problem. I originally chose it for 8x SATA, but since I’ll go with 3x16TB, 6 should be fine too. Normally I’d go with ASUS, but their boards don’t have enough SATA. Intel NIC also is a selling point definitely. The only thing that’s annoying about the Z690-A is that they don’t list how the 2 additional 16x slots are wired, I’m pretty sure they aren’t actually 16x, from the looks of it they are 4x.
I’d consider G.Skill pretty reputable, they might even be the biggest brand in that space, it’s just their low end RAM though (They are normally known for their TridentZ memory, but that would be a waste for a server like this and twice as expensive).
Nah, the case I found is basically just all mesh and drive cages, ideal. The define series is too expensive and has too little Airflow. Love their cases, my desktop enjoys the meshify, but for something I’ll just dump into the basement it’s a total waste.
980 should be pretty premium already, I could go with a 980 Pro, but for an OS disk and cache I think it’ll be more than sufficient for now. (I mean I won’t say no if it goes on sale though)
I’ll probably go with a be quiet pure power, it’ll be pretty overspeced anyway and has no spiky GPU, so I don’t think it’s fine to go with something more regular. That stupid configurator on has some kind of problem though, it won’t show half of the PSUs, maybe I’ll need to redo it in PCPartpicker again…
G.Skill is also known to have a issues with their Trident series so it depends on how you look at it. I might be wrong on this one but there doesn’t seem to be a way to do RAM within EU either :-/
All mesh and cages doesn’t mean it’s decent, you do want to care about structural integrity (vibrations) and crap like razor sharp edges. Most decent cases also comes with fans so you don’t need to buy additional ones. I have Design XL (v1) and it works fine as a server without any issues at all.
What I mean by the 980 is that you pay quite a bit because it says Samsung, P5 Plus performs quite a bit better.
I would be careful with the lower end stuff, PSUs do usually fall under the you get what you pay for category. The RM series are also usually on sale so that doesn’t hurt.
Please port what you end up with in the end
As for the distro vs FreeNAS, you most likely want to keep at least the multimedia stuff pretty much up to date and stuff like that moves much more slow on such a target as it’s not a priority. SMB and NFS etc depends a little bit on the distro, I can only say that NFS sharing is dead simple in FreeBSD and I’ve never gotten it work on OpenWrt (I suspect it’s a bit quirky on OpenWrt as there are a lot of posts about it) but I haven’t tried any other distro. It’s no that hard to be honest and can be quite useful information further down the road.
Well don’t care about that pretty much because I’ll build it once an then never again if it weren’t for the disks and me potentially destroying some thing when I hit it, I’d even consider just putting the motherboard on a shelf without any case at all, it literally only has to hold the components in place. Recently built another server in another cheap case and while they aren’t great, it was totally fine for just shoving it up a shelf.
I’ve used TridentZ in my last 2 builds and never had a problem.
Well, the 980 is 100€, the P5 Plus is 130€.
Well be quiet is pretty reputable and the Pure Power is not their lowest end, but more lower midrange. Had a lack of EPS connectors anyway though, so it’s a seasonic focus now that also has higher efficiency (Platinum), always good for a server.
Well as I want to go with TrueNAS Scale, multimedia stuff won’t be a Problem as I’ll just deploy it with docker. SMB and NFS should also be no Problem, the only issue is having a web interface, but that’s basically no matter the distro, I’ve only seen it on consumer grade NAS devices up until now sadly, not even something commercial like unraid has something similar. As for the OpenWrt issues, that might be a problem because it’s mostly targeted at router AFAIK and not NASes, so I’m surprised it even has NFS support.
I’ve also updated the list with PCPartspicker so the correct PSU is in there, etc.
In all honesty, the best part I like about ZFS is deduplication. It is slow, so without RAM, it’s not very good for live activities, but if you have the RAM, using deduplication to save storage from VMs that have the same OS installed is awesome. If you have 20 Rocky Linux VMs, each using a base 8 GB, you can save 160 GB. In an enterprise environment, this gets even bigger. For a small business, I used to manage 250 VMs and I never got around to implement ZFS for real, besides one single server that didn’t even need it. I could have saved 2TB of disk space easily with just the base images, but many VMs were clones after each other, some being between 200GB and 400GB large, with less than a dozen being a few TB large. I would have probably had savings in the 10s of TB with dedup. Aaah, what a waste.
But more importantly, in case you lack the RAM, you can still use dedup for backups. You can realistically have 12 full backups (1 full each month) not using more than the incremental backups in-between them. I never hear people talking about ZFS for backups. Well, I never hear people talk about backups in general, which is a shame.
Back to planet earth, I really want to finish my small NAS build and show either how efficient one can get with a build, or what a terrible mistake I made (it can go both ways). But worst case, I’ll still have a device ready to go for network backups.
Well first we need some proper backup software, everything I’ve seen is either to complicated or it’s synology’s software, so it only works on their hardware. Maybe a devember project or so… I just want something that has a core service, source agents and target agents and a nice web UI…
About de-duplication, apparently you can get around the memory issues with the special metadata device. It’s probably slower than in-memory (because it has to search the SSD), but it should be easier on the memory.
Disk space is one aspect, but this also applies to cache aka memory. With these kinds of dedup ratios, you can cache 250 VMs at the cost of the compressed block size of the OS and dedup tables, which is GBs of memory for TBs of data.
This is a special use case however. For a homeserver and most other stuff, enabling pool-wide dedup is a recipe for disaster. The rule and advice “just don’t!” still applies.
Regarding backups…well yeah, dedup works. But why use dedup when you have access to snapshots? Far less headache and less overhead.
zfs list -rt snapshot rpool | wc -l
on my laptop shows 1674 snapshots in total, all done fully automated. That’s how cheap and easy backups are today. Combine this with zfs send/recv to replicate snapshots to another destination (backupdrive,NAS,cloud,whatever) to make it a real backup. I’m a lazy person and I had more trouble getting backup to work with commercial backup solutions on windows, and they all took ages to complete. Having TrueNAS at home, you can do it with a couple of clicks in the GUI too. Sending your /home@ incremental snapshots directly from laptop to NAS is so convenient.
And about that Samsung 980…I have that model running myself on the laptop and I like it for having very good power states and being very power-efficient, which is also important for a server that runs 24/7. Above average endurance and solid PCIe 3.0 performance. Would buy again. A shame they don’t sell 2TB versions.
The best way to get around memory issues is to get more memory. Or optimize work load to use less memory.
Well of course, if you have an unlimited budget. But for a home server sometimes substituting an SSD for 100s of GB of RAM might be the better option.
Just keep in mind that both snapshots and local copying aren’t backups, but more or less snapshots, you definitely wanna keep copies of you most critical data in another (ideally off-site) machine.
Well the good thing with SSDs is that you will know they will fail soon enough normally, so as I can forsee my financial situation improving in the next year’s I don’t even care that much as long as it hold 2-3 years. After all if the cache fails it fails, not much to it.
Well I still think there’s something missing, having the option to do it via CLI is cool and all, but I’d really like some proper solution, which apparently just nobody has built (open source) for some reason? There are things like bareos, but they are just so awful to use… I mean not every source is ZFS too, I wanna be able to just stream the contents of my PostgreSQL database into a backup location somewhere.
Might make a lot of sense for me at some specific places though. Turning it on in the right places is the important thing. Sadly I don’t see an option just run deduplication manually for some files once… That would be useful, to just dedupe your cold archive once in the month/year and not constantly on the fly, I think btrfs is able to do that.
Since I haven’t been able to get the hardware yet (although I might be getting the first stuff this month), the 13400 & 13500 look nice… Depending on the price of course, but some E-Cores can’t hurt, I’ll definitely have a use case for those…
With AMD releasing AM5 now, another use case has opened up itself. All AM5 chips will have some level of integrated graphics; this means you can hardware encode/decode without a GPU. This is potentially something to consider, although, cheapest platform right now still costs a fair bunch.
Here is a revised setup with a 7600X based on the one I recommended above, it is pricey though, not something I recommend right now and I think this will be around 10-15% cheaper system price within six months - at which point it could be an option, especially if AMD release a 7600 or 7500. Just want to explore all end points.
I still recommend 12100, 12400 or 13400 assuming pricing stays the same - However, if the 5600G + Motherboard is less than 12100 + Motherboard, get the 5600G.
