Looking for advice on Truenas setup- Pools,Cache,Specials

You can test this by letting the pool run with sync=disabled. If it’s faster (disks will tell you), you will benefit from a SLOG. Use your P1600x in this case, they make a great SLOG.
If it’s not faster, SYNC writes aren’t a thing and you don’t have to worry about random writes. You may want to test this with the NVMe pool as well. Flash usually performs the worst for random IO, so you want some way to avoid it.

If you can afford all those disks, 1-4 TB NVMe for mirrored special vdev is barely noticable in costs. No need to limit yourself to 100GB and make things complicated. And with more space to work with, you can include small files too. Bandwidth isn’t a limiting factor, so PCIe 3.0 or even SATA will be fine.

Correct. But it doesn’t work the other way. ZFS doesn’t move the stuff off the disks to fill a newly added special vdev. Just like it doesn’t balance the drives when adding a new mirror for data.

Check out my guide for general usse and recommendations on ZFS and support vdevs: https://forum.level1techs.com/t/zfs-guide-for-starters-and-advanced-users-concepts-pool-config-tuning-troubleshooting/

1 Like