How much ram for TrueNAS Core?

I heard the rule of thumb is 1gb per tb of raw storage. I have 24 TB drives with eleven of them so 264 gb. Does that number include cache drives or just the HDDs in the zraid?

What I’m doing is building out a dual socket build, and adding the second cpu later, after benchmarking with one cpu, and figuring out how much I need in terms of SSDs and what types of SSDs. After that I’ll try out different arrangements of spreading the disks across both cpus to see which maximizes throughput and bandwidth.

I’m having a hard time finding kits with 8 sticks above 256 gb. After 384 gb. I’d have to go all the way up to 512, which seems excessive. I guess when I add the second CPU I’d be using two 256 gb kits and flipping this ram. Is 384 enough?

G Skill has some nice 8 stick 48 gb kits. Based on pricing it should be a lot faster than competitors for about the same price or slightly lower. The only problem is finding those kits. I could purchase two four stick kits, but who knows if those will be reliable together. Anyone know a good place for buying server memory? Am I pretty much buying directly from the manufacturer above 256gb?

1 Like

Clear case of not understanding the matter.

When it says “rule of thumb” that doesn’t mean “absolute rule that must strictly be abided:roll_eyes: Your system will run fine on 256GB RAM and probably will get by with just 128GB as well. As you’re planning on adding a 2nd CPU, just buy 512GB and be done with it. No need to flip it, just expand. RAM will work on the slowest speed stick anyway.

2 Likes

I’d rather overspec the machine, and have performance head room for whatever I throw at it. I don’t believe 256 gb is enough for using dedup, and adding cache drives. Ideally I’d have 24 sticks of memory, but I might be able to avoid that for awhile.

I would just start with 96G and see how it goes

2 Likes

That rule is stupid and wrong.

You will most likely not be CPU bound if you don’t use some exotic compression or encryption.
If you use SMB, single core performance is important. But even then, my old atom can handle 10GBit/s. What speeds are we talking about?

Depends on what you wan’t.
ARC uses has much as you throw at it. It never has enough, even if you have 2TB RAM :joy:

TLDR: Describe your workload.

Depending on what you need, there are different approaches.
Sometimes you need a SLOG. Sometimes you need a faster pool. Sometimes special vdev could help.

Most of them can be added later on, AFTER you ran into performance problems. Many times, the out of the box performance without any extras is decent enough. Special vdev is the only thing I can think of that is really annoying to do afterwards and should be done at the beginning if needed.

1 Like

If you are not running de-duplication (and don’t) this is not a rule of thumb.

It’s a rule of thumb to hold the de-dupe table in RAM to avoid performance while running deduplication falling off a cliff. essentially, every write you incur when running deduplication needs to be looked up in a sha256 hash table of every unique block on the pool, to see if the same block already exists in the pool. To avoid flogging the shit out of the disk, this table is held in RAM. That’s where the rule of thumb of 1GB per TB comes from - it’s a rough estimate of the unique sha256 hash table vs. the actual storage blocks the hashes represent.

As you can see, if the table doesn’t fit in RAM, every single write to the pool would potentially need a scan through the table on disk…

Don’t do deduplication (it’s pointless in most cases, especially for a home lab, etc.)

Without that, ZFS is fine with a small amount of RAM. More is better for cache, but you aren’t going to run the machine out of RAM unless you’re running dedupe.

You could get by with 8 GB - but if you’re building something to be fast just consider how big your hot data set is and how many random writes you want to absorb in cache that may be coming in faster than ZFS can serialise them to disk.

5 Likes

I’m building this out for backing up up to 100 computers, media streaming, caching packages. Probably some other data storage as well. I’ll use it for distcc as well. (mainly why I’m getting a dual cpu board.) But, this machine is mainly for backing up the other machines. I’m going to have to figure out the power needs when I get to that point.

Going to start off with five and add more over the first year. I mainly need a backup solution that can restore entire systems quickly with minimal down time.

I’m probably going to be using NFS. I think theoretically with bonding I should be able to get up to 20 GBit/s. Maybe, higher not sure what my SFP cables can pull.

As far as, I understand the main benefit of dual cpu for storage is faster access times by splitting drives between both cpus. Mainly trying to avoid thrashing if I have one machine backing itself up, while another is pulling data from the drives. I need to be able to restore machines as fast as possible if I have to take one down, or worst case I have to update all of my systems on a staggered schedule.

If you’re using NFS, be sure to tweak the ZFS options for “noatime” - as otherwise you’ll get a last accessed write on the files when they are read.

This is a VM on NFS performance thing, but will probably help in your instance as well.

Sounds like your workload MAY see benefit from dedupe but these days, disk is cheap, compression works pretty well and the RAM/speed cost for Dedupe is (usually) not worth it. Once you’ve got some live data there’s a command to estimate dedupe savings that you can run and then you can assess the trade-off.

In addition to the hash table size needing RAM - the lookups need CPU too.

lz4 compression on ZFS was described some years (like, 10) ago as “basically free” as modern CPUs are fast enough vs. disk speed to make it very much worth it. so turn that on too.

Yeah, I’m planning on getting the system up first, and then running all of those tests. Just getting a dual cpu board so I don’t need to pull the mobo if I end up finding out I have a use case for the second cpu. But, I’ll probably eventually add it anyways to get the core count up on my network for compilation tasks.

1 Like

The 1 gb per tb rule of thumb isn’t dumb or wrong, but like most oft repeated things, it lost it’s original context. Minimum required system memory | TrueNAS Community

The more memory you give ZFS, the happier it will be. Maxing out your memory is the first step before moving on to things like L2ARC, etc.

Your pool layout will have a large impact on your speeds as well as the types of files transferred. You may need to focus on IOPs instead of bandwidth. How do your backups work? They may not be able to max out a single connection.

Another consideration is breaking out the different tasks instead of attempting to have one server do everything. That would reduce your requirements per machine and help avoid opposing needs.

Lastly, why Core instead of Scale? Core is going away and is unlikely to see any more updates.

The backup part is a black box. This really depends on what software you use.
Media streaming and caching packages on the other hand is mostly sequential reads.

20GBits is pretty easy. Again my old atom can do 10Gbit/s without much sweat over NFS.

Not really.

Then you need a faster pool or limit your backup task.

Trust me, you don’t. Certainly not with just two SFP+.

Since money does not seem to much of a problem, I would either go with two six wide RAIDZ2 and a special vdev 3 way SSD mirror.

If not that much storage is needed, I would go with six mirrors and a special vdev 3 way SSD mirror.

If your backup software or anything else in your workload issues sync writes, I would get a PLP SSD as SLOG.

1 Like

I have hourly, daily, weekly, and monthly backups. I can’t remember the exact amount I typically retain for each. I currently am using Borg to dedupe.

Pretty sure, the devs behind OPNSense are forking Core. One of the guys that posts frequently in the Deciso forums hinted at it on one of the old TrueNas forum. I also believe Core performs slightly better for backups.

I’m not familiar with Borg, but it sounds like you won’t have sequential transfers. What performance are you seeing with your existing setup and what is it running.

Big fan of the OPNsense folks, but I wouldn’t trust my data on a possibility. Also, if you use TN as storage only, you’ll be able to fairly easily switch if needed. Hyperconverged servers get messy fast.

And if you set the RGB to red it goes even faster. /s

In all seriousness, you need to figure out what problem you’re trying to solve. Are backups taking too long, are you running out of space? etc Find your current pain point and do some performance testing to figure out the actual bottleneck and then you can do some small scale POC testing to determine what changes have what effects. Otherwise, you’re just wasting money because you’re twiddling unlabeled knobs hoping for a good outcome.

One thing I forgot to mention from the OP. When people think of cache drives, they’re usually imagining something like how Unraid does things, which is not how ZFS works. Incorrectly adding SLOG or L2ARC will result in at best, no benefit. Worst case it makes everything slower or causes you to lose your data.

3 Likes

This is what we run in enterprise deployments, what is your planned application?

It’s more specifically 1GB per TB installed + 8GB for HostOS during updates (+ 5GB per TB installed if using deduplication)

If you need to stretch it, bring it down to 1GB per TB of usable storage.

With 11 drives, I am assuming you will run all 11 in a single RAIDZ3 vdev?
That gives you 192 TB usable and decent resiliency.

edited to add:

Having built enterprise backup solutions for this exact application, you’ll want dual cpu’s or at least second gen epyc and 1GB per TB installed should be your baseline.

40 gig uplink to the switch minimum, likely multiple 25 gig or a 100 gig uplink.

That much throughput will require massive RAM to hold the data being written to disk as you’ll quickly outwrite the drives (8x 24TB CMR drives are currently running 1,600 MB/s ideal).

Additionally, you will need cache drives.

This is an enterprise scale deployment.

This workload is not trivial, meaning you probably have done at least paper napkin level math on how you want this to work - including a scale plan which provides soft/hard limits for this platform.

Looks like this is highly io bound (network and storage), so not sure where the dual CPU requirement stems from unless it’s driven to support suffient IO via the additional PCIe lanes such a platform can provide.

Backup vs backup vs backup.
The question you ask is about RAM requirements for ZFS. There is very little unless you dedupe (as said above). After that it’s all basically a fancy cache.
I like using ZFS snapshots combined with zfs send/receive as a backup strategy for as many apps as I can. It is conceptually easy and very efficient. Is this what is going on here (I think I unintentionally projected by use case onto yours)?
Maybe additional backup software? mix and match?

Ditto on the BSD vs Linux debate: I think in 2025 BSD needs to be labeled legacy or at least future-risky due to the unbalanced amount of resources being invested in either platform. TrueNAS has basically stated their future is on Linux and any new deployment into core is hard to understand.

3 Likes

I back up my entire system to one disk, which I’m able to dig through the history for files. The current bottleneck is the USB 3.0, or more accurately the speed of an HDD. I’m building a more robust back up solution.

First time probably takes 15 to 30 minutes for the initial backup, and 6 to 8 minutes for each subsequent back up. I’m going to be doing this with five machines initially, and adding more machines over the course of a ten year period.

What takes forever is restoring data, and searching for files. I’ll probably be using ZFS for atomic backups and borg for backing up my data on here. I’m going to have to figure out if turning on dedup for ZFS speeds up borgs deduping at all. (maybe, if multiple machines have matching byte patterns) Looks like Borg is adding parallelized workloads by the time I finish this machine. Pretty sure, I could back up multiple directory structures using borg create in parallel now. But, I don’t know if I’ll even be using Borg for backing up on this machine.

I believe FreeBSD benchmarks faster than Linux for a NAS if you have the right NIC. Everything I’ve read about both operating systems is TrueNAS Core is still better for a system that’s mainly a NAS, and there are plans to continue updates for a few more years. At least, according to the people using Core, and the devs talking about forking the project. TrueNAS Scale sounds better for a more general purpose server that also needs a lot of storage.

My budget is basically $20k to $30k with me trying to come in closer to $20k. I’ll spend closer to $30k over multiple years of adding upgrades to increase performance over the years.

I’ll probably be using Borg with ZFS saving atomic snapshots. (in case I delete my borg archive) The idea behind dual cpu is just for IO, and adding more cores to my network for compiling. Since Borg currently supports backing up to a repo with multiple threads crawling different file structure that might be a benefit.

Maybe, have each cpu crawling through a systems directory tree saving to their own vdev. But, that would be long into the future. I’d probably need a vdev for the second cpu. Oh, this could get crazy expensive, but be really fast at backing up and restoring stuff.

That should help with accessing files in backups, and also having this operate as media storage, and a package cache. Might end up tossing other files on here as well.

I do know I’ll be building a second machine, eventually, for IPFS, and that definitely benefits from dual CPU.

I’ll probably be turning dedupe on to see if it speeds up Borg. I’m guessing it might if two systems share a byte pattern. It’s my understanding that before adding L2ARCS, Zil / Slog, or a cache drive the recommendation is max out ram first. I’m probably going to have one 386 kit of 8, and maybe two. I’m guessing putting the second ram kit on a separate CPU doesn’t count as mixing kits. The G Skill server ram seems as reliable, faster, and cheaper than other options. I think I’ll end up having 4 empty ram slots on each cpu since they don’t make 12 ram kits, and I’d rather save money on that.

I’m confident it’s going to get forked as I’ve seen some devs make some vague statements about getting the necessary processes in place for starting their own NAS company. But, that’s like one to two years out. They have an email and website. It’s Deciso doing it most likely. This guy being involved is promising. Probably, hiring people right now, and maybe I can get a job building systems for them in the US to avoid tariffs.

Why? Even a single CPU can handle 100GBit/s.

Well, it depends on what you want.

Basically for writing backups, we have to know if Borg uses sync or async.
And if SSH could be a bottleneck.

Assuming it is sync: Add a PLP SSD as SLOG
Assuming it is async: Nothing you can do, besides building a faster pool

For reads or restoring the backup, the 12 wide RAIDZ2 should be more than sufficient.
But maybe metadata performance also plays a role here. Maybe borg checks a lot of chuncks or compares them for compression. In that case, two very cheap 512GB SSDs as special vdev will do wonders. Or at your budget, even three in a three way mirror.

So for a decent start, I would use special vdev and two 6 drives wide RAIDZ2 and a SLOG.

Then 64GB or even 32GB RAM are probably fine. Sure, more ARC is always better, but probably not needed that much in your case.

Dual CPU is total overkill. I would rather check if you can get a 100GBit/s QSFP28 NIC or at least the now dirt cheap SFP28 with 25GBit/s.

In general FreeBSD does have an edge over Linux when it comes to disk I/O however the 13.X-branch is getting old and may cause issues if you’re planning to use fairly recent hardware. That said, you’ll be on barebone FreeBSD if you want to go 14 or newer at least for now which isn’t much of a hassle.

Depending on your requirements you might want to consider ECC UDIMM unless you’re referring to R-DIMM. That said, I would stick to JEDEC specs on models from Kingston, Cruicial/Micron, Samsung, SK Hynix if reliability is a requirement and stay away from brands that do not produce everything in-house.

As others have said start without SLOG / L2ARC as it probably wont help much in your case (as long as ARC is reasonable sized). Given the rather vague description you’ll probably be fine with 128Gb and have quite a bit of margin to spare. A metadata cache might however be beneficial depending on your workload as mentioned.

I would be very careful with dedupe, why do you think it would help?

Another thing you likely want to keep in mind is time to resilver, that array is going to take very long to resilver which may not be ideal if the data is important.

I’m also a bit curious about TryTwiceMedia’s “recommendations” as they pretty much seem to be the same regardless of scenario. I would also look into a beefier NIC especially if you plan to serve media from it and go for a CPU that has uniform cores.

As for media streaming, you likely want to look into nginx or possibly mediamtx. I would strongly suggest that you look into other solutions than distcc which is dated by now.

2 Likes

I know Borg uses SSH for remote backups I’ll probably be mounting this with NFS, though.

I believe currently Borg is mostly synchronous writes. But, they’re improving multithreading support, which might make it asyncronous. I’m asking the dev to find this out.

How Borg works is it stores a local cache of checksums to chunks of backed up files, and consults these for deduplication purposes before compressing the data. The entire cache has to be rebuilt if multiple systems access the same repository. So, it’s one repository per computer.

I might actually save money with dual cpu. I’m running distcc on my network anyways. So, that’s one less motherboard and chassis to buy and probably some other stuff as well. So maybe, I save $500 to $2k.

But, yeah I need to be looking into those higher bandwidth network cards. Might need to upgrade the switch on my network to handle that.

What’s wrong with distcc? There is icecream, which is easier to configure. There is incredibuild too and SNDB, but those are proprietary and expensive to add more computers. I should still see build speed ups.

Chromium takes 40 minutes to build even with 80 cores compiling it on the local machine. Depending on how localized the data is having ten machines compiling it should still speed up the compile time.