Seeking advice on TrueNAS Scale vdev layout (I know, cringe)

[mrsuspenders15]

Hi all!

After many months of research, saving up, scouring ebay, and collecting old parts from workstation builds, the time has finally come to build my first ever server for my burgeoning homelab! I’m nervous, excited, and in need of some last-minute guidance. First, the context.

Purpose

In order of priority:

1. A network location to store large source video files to point to in Davinci Resolve (my/my work’s video editor of choice) on my personal machine.

• Multicam video shoots with 1-3 4k feeds and 2-6 1080p feeds (depending on the shoot), to be deleted after ~3 months

• Lengthy (1-3 hours) 1440p, and 1080p livestream recordings, to be deleted after final edit

2. Hosting game servers (Minecraft, Valheim, etc) for my friends. The networking side of the homelab will probably be a separate post, but I’m going with a Cloudflare tunnel to expose services to the boys.

3. A digital playground for me to get some hands-on learning with contanerization and VMs. Portainer, Linux distros I haven’t tried yet, and Windows server, as a start.

4. Hosting a Jellyfin server for my household.

5. Doing other cool things in the future, whatever they may be!

Hardware

• CPU: Ryzen 7 3700x

• Memory: Crucial 2x32GB DDR4 2666 RDIMM

• Storage: 8x8TB Seagate Exos HDD, 2x256GB Teamgroup SATA SSD, 1x500GB Crucial PCIe 3.0 NVME SSD (there are exactly 8 HDD enclosures in the case, so 8 is my limit without getting a different chassis)

• Mobo: ASUS Tuf X570 Plus WiFi

• GPU: EVGA GTX 1080 (my beloved)

• Network: 10Gbps LAN, 500Mbps down 20Mbps up WAN

Software

• TrueNAS Scale as NAS device and Hypervisor, with the understanding that I may eventually outgrow its hypervisor capabilities and need a different dedicated VM host in the future

• Snapshot backups to my existing Synology NAS (which I know is only 2 copies of 3-2-1 but I haven’t found a compelling remote backup option yet)

---

Thanks to Wendell, Jeff, Tom, Other More Differnet Jeff, Linus, and many other creators in the tech space, I’m up to speed on the basics of ZFS/TrueNAS. Without their videos, I’d be much more apprehensive about the do-it-all setup I’ve undertaken. However, I still have some outstanding questions.

How should I lay out my VDEVs?

I know, every noob asks this. I’ve looked online and haven’t found a consensus. Based on what I’ve read, for my current setup I have 3 options

1. 4 mirrored vdevs

2. 2x4 RAID Z1 vdevs -everyone disliked that

3. 1x8 RAID Z2 vdev

I’m leaning 1x8 RAID Z2 because, despite the read/write performance hit, I still feel hinky about the single disk redundancy of mirrored vdevs. I know you could potentially lose one drive per vdev, but with my luck if I lost 2 drives they’d definitely be from the same vdev. 50% storage efficiency is also expensive imo.

I’m willing to be convinced otherwise, though.

What do I do with the 2 SATA SSDs?

My current plan is to install TrueNAS on the NVME ssd (I know 500GB is overkill for the OS, but I inherited the drive ). I was wondering about either a L2ARC or SLOG configuration for the remaining SSDs (striped for L2ARC, mirrored for SLOG), and which would benefit my use case the most.

---

PHEW, that was a lot for a first post. I’m so glad to have found a community of people who clearly know way more than I do and from whom I can deepen my understanding of computing beyond the “magic box go brrr” level!

[cowphrase]

@mrsuspenders15

After many months of research, saving up, scouring ebay, and collecting old parts from workstation builds, the time has finally come to build my first ever server for my burgeoning homelab! I’m nervous, excited, and in need of some last-minute guidance. First, the context.

Congrats, it’s a nice moment when you get together your first system.

The first thing I’ll say is that you’ve got a lot of purposes thrown onto one piece of hardware. This is fine, but the differing work loads will reduce performance for others. The biggest point I’ll highlight here is to try, and see how you go. You won’t know there is a problem until you encounter it.

Will you be doing video editing off this NAS using Davinci Resolv (using SMB or NFS?)? If so the seeking performance will be important. Seeking means random read IOPs, will be vastly superior on 4 mirrored vdevs. Minecraft and game servers will also want a lot of random read IOPs.

However in terms of space, 4 mirrored vdevs is a large waste. And your live streams / jellyfin media will be just fine on a 1x8 RAID Z2 vdev.

I would propose the following:

500GB NVME for TrueNAS OS drive.
2 x 256GB Teamgroup SATA SSDs as zpool mirror
8 x 8TB drives in RAIDZ2

Alternatively you could setup a 2 x 8TB mirror pool, and a 6 x 8TB RAIDZ2 pool. You’d have more zfs pools to manage, but it helps separate the load.

You can host your minecraft server and put specific files onto the SSD mirror, which will improve read performance a lot. And you should test to see if your editing performance is fine on the RAIDZ2. Important point is to test before committing - you can always reformat the array.

Another point I’ll make is that L2ARC and SLOG are rather situational. L2ARC helps with work loads that re-read a lot of data within a short period of time (eg: databases with huge indexes), but before you setup L2ARC you buy more RAM. SLOG helps specially with SYNC Writes. None of your workload is going to benefit hugely from these, and SATA SSDs aren’t the best for these anyway.

[mrsuspenders15]

@cowphrase

Important point is to test before committing - you can always reformat the array.

Thanks for the helpful tips!

I’m going to go with a mirrored pool for the SSDs as you suggested, and I’m going to try a Raid Z2 for the HDDs, and test the editing performance over SMB with an example 4k project (we fortunately have one in the hopper, so the real-world test is ready to go).
If the timeline performance with random seeking is acceptable I’ll stick with it, otherwise I’ll suck it up and go with 4 mirrors, since Resolve timeline performance is the primary purpose of the server.

[H-i-v-e]

@mrsuspenders15

I’m leaning 1x8 RAID Z2 because, despite the read/write performance hit, I still feel hinky about the single disk redundancy of mirrored vdevs. I know you could potentially lose one drive per vdev, but with my luck if I lost 2 drives they’d definitely be from the same vdev. 50% storage efficiency is also expensive imo.

In my opinion you could also go with the 2x4RAIDZ1 setup too as long as you setup monitoring/notifications so you will be notified soon after a drive failure and as long as you make sure you backup soon and often to the other NAS. While having said that, me personally, I would have gone with the RAIDZ2 as well.

Same advice that goes for the RAIDZ1, having to setup monitoring of the arrays health, should go with the mirrored setup as well. With RAIDZ2 and the double parity you have a little bit more slack.