I’m considering a redesign of my TrueNAS homelab NAS and would like to get input on the theoretical best approach. I understand that the only way to know the best configuration is to test with my workload but that is quite hard to do as I do not have an extra test NAS. I am primarily looking to see how to improve performance for VM and homelab selfhosted services storage.
Current Setup:
ZFS pool with 2x mirrored vdevs (no SLOG/ZIL, special metadata vdev, or L2ARC)
48GB RAM (planning to upgrade to 64GB soon)
10G networking to one Proxmox host, 1G to the Mini PC PVE host
Stored data includes:
Personal files / documents
Family photos & videos
Media library (video files)
VM disks
Backups
Backend storage for homelab services (e.g., databases, Docker volumes)
What I’m Debating:
Option 1: Two Separate Pools
HDD Pool: 6x HDDs in a RAIDZ2 vdev (for documents, media, photos, etc.)
SSD Pool: 3x SATA SSDs in RAIDZ1 (dedicated to VM disks and homelab services)
Option 2: Single Pool with Performance Optimizations
6x HDDs in RAIDZ2
Add one or more of the following:
Special Metadata VDEV: to accelerate metadata-intensive workloads
L2ARC: assumedly would cache VM disks since my board only supports 64GB ram
SLOG/ZIL: for sync write acceleration - (VM disks are all via NFS which is sync=always)
My Goal:
Both options would handle docs, photos, media-server data just fine. I’m trying to figure out which option will give me better performance for homelab services, essentially VM disks and Docker database storage. I’m also open to hearing if other setups might make more sense based on your experience?
Looking for feedback on:
Real-world performance differences between the two
Whether L2ARC/SLOG/special vdevs on a HDD pool are worth it in practice compared to an SSD zpool when my capacity need is only about 2TV
Storage size context:
docs/pics/media server/backups - approx 25-30TB
VMs/HomeLab stuff that needs performance - approx 2TB
3-2-1 Backup:
As a note, I have an offsite TrueNAS that ALL data (except media server) is backed up to, regardless of the storage layout I am debating above. A subset of my most critical data is also backed up to a cloud provider.
TLDR - Can 1 vdev of 6x HDDs in RAIDZ2 with enhancements such as special metadata device, L2ARC, ZIL/SLOG beat the performance of a basic SATA SSD zpool with 3x drives in a RAIDZ1?
I was running Option 2 for 3 years on my server (technically pool is still there, not migrated yet). 96GB ARC, 6x 16T HDDs in 3 mirror vdevs, with 1T special vdev and 1T of L2ARC. Smooth and performant…I eventually removed the special (wasn’t doing much really to justify two drives) and added the drives to L2ARC which had 2T total during the last year.
Memory → tuning → Memory → L2ARC → tuning ->>> special.
That’s my priorities that work the best for me, having a pool with a lot of (mostly 8k blocksize) zvols on the one hand and several datasets with large stuff on the other side. So it’s basically your average homelab pool.
I never used a LOG cause I went for pool-wide sync=disabled because I’m a cheapskate and it is faster.
4+2 RAIDZ2 is a nice topology for a RAIDZ. But I never liked RAIDZ, too inflexible, too slow, too clunky. So don’t ask me to vote for any RAIDZ. I will always advocate for mirrors, or 2+1 RAIDZ1 if I’m generous.
ZFS is about “can we cache stuff so it isn’t slow”? That’s the ZFS approach that can mitigate wide vdevs, HDDs, fragmentation, random IO, and all the other stuff that is bad. And why ZFS is great and HDDs feel like NVMe if you do things right.
Option 2! Go for the ZFS philosophy, one big pool, make slow storage fast by software, DRAM and supporting drives.
Interesting, this was actually the opposite of what I had guessed folks would recommend.
Also good to note that you are using mirrored pairs and not RAIDZ2.
I currently use mirrored pairs and store my media collection on Synology. (The storage efficiency of mirrored pairs isn’t worth it for media storage). My thought was to get rid of the Synology and have a singular NAS using RAIDZ2 for a balance of redundancy, speed, and storage efficiency. (RAIDZ is especially compelling with 1- the new RAIDZ expansion feature and 2- the upcoming ZFS rewrite subcommand that IxSystems is developing)
My thought was that as SSDs become so cheap, and my capacity need for fast storage is reasonably small - the best solution is Option 1.
That said I really like the simplicity of your solution, I’m worried raidz2 even with all the enhancements I throw at it will have poor performance for VM disks.
This would be a fun project to test if I had an extra NAS lying around with a bunch of drives!
I think it really depends on how many ports you have.
For the VMs, I would go with a 3 way mirror, instead of RAIDZ1.
SLOG could really help for sync writes. But at the same time, you might not have many writes?
If you can spare the two ports for special, do it! Metadata performance even on two old SSDs you have collecting dust, is like night and day.
I have a HBA card that is mostly unused so I have lots of free SATA ports, can’t recall specifics off hand.
If I went the 2 pool route, I’d be open to using RAIDZ2 on my HDD pool and doing mirrored pairs on the SSD pool. I see no benefit to a 3 way mirror. The SSD pool is subject to 3-2-1 backup process, and I do not feel the need to incur additional costs for uptime sake.
For my needs, I’m not sure if SLOG is beneficial on an SSD pool. I’ll need to research that more
On this point, similar to the above- I don’t know if a special vdev is worthwhile on an SSD based pool. Seems redundant to me? But I’d need to research more. That said, I wouldn’t want to use cheap or old SSDs here given that SSDs typically have a lower write endurance than HDDs and if I lose the special vdev, I lose my entire pool.
I thought you have 3 already.
You can a little bit faster reads by using a 3way instead of 2 way mirror.
SLOG speeds up sync writes. If you don’t use sync writes, you gain nothing.
Does not matter if SSD pool or HDD pool. PLP SSDs can lie about sync status (because they have capacitors to write down stuff in an emergency, which is why they are faster for sync writes.
No, not really. SSD pools have fast metadata performance already.
Yes, but also reduces number of vdevs for a given amount of HDDs. Pool read speed should be constant assuming even data distribution among vdevs. You increase redundancy at the cost of storage efficiency 33% vs. 50%. Read performance is pretty much a wash and write speeds are lower on a pool with 3-way mirror.
