I’m about to assemble my Storinator Q30 tomorrow. I’ve got 16 of the 14TB exos drives. I have an Optane 905P NvME stick. The machine is, I think, a SuperMicro 11XSPL with a Xeon Silver 4210 and 128GB of memory.
I’ll be moving some light virtualization to it and editing 4K video in Final Cut from a Mac. Single editor, but backups can and do happen anytime and take some bandwidth too, so it’s not truly single user.
I’m planning to run TrueNAS Core on top of Proxmox.
I was thinking one big ZFS2 storage pool along with the Optane as the ZIL and L2ARC.
Instead of an L2ARC, maybe consider a ZFS Metadata Special Device: Z . It essentially moves all of your metadata onto (ideally a mirrored pair) of SSDs.
With 16 drives you have a high chance of a second (or third) drive failure when you’re resilvering after an initial drive failure. I’d split those 16 drives into two RAIDZ2 pools of 8 drives, or switch to RAIDZ3. You may also want to consider a hot-spare.
Sounds sensible. You probably want to arrange the drives into multiple RAIDZs - containing 6-8 drives.
There will be good use for the Optane. Which one depends on your use case.
Consider use as Special Device. Also, and I don’t know if this is possible in TrueNas, you could partition the Optane and configure alls three use cases on one Optane drive.
I’m just going to chime in here early but depending on the level of redundancy this could get complicated
Do not have special devices or ZIL That is less than your equivalent level of redundancy or you will lose data when they go down
So if you do raid Zed 2 and you want to do these things, make sure you have two levels of redundancy for each of those special devices. That’s just my two cents
It’s not that the data is going to be gone in most cases. The data is still going to be there. It’s just going to be a giant jigsaw puzzle you have to put together and it’s not very fun. With l2arc youll just lose a cache no biggie. With zil you could lose writes on power loss… kind of a biggie…with special devices. dont shoot yourself in the foot
L2 Arc, you’re probably okay. It doesn’t matter too much and it doesn’t really give you that much speed improvement unless you have a lot of asynchronous reads. Same with ZIL But for synchronous writes
I think before considering anything, it’s very important that the original poster (@Dave_Plummer ) understands a lot more about ZFS so he knows the caveats and trade-offs he’s getting into.
What’s the need for truenas on top of proxmox? Proxmox understands zfs natively and you can probably get a slightly newer version of samba from the proxmox side of things.
If you have more ssds a mirrored/redundant special metadata vdev will work really well. You may not really even need l2arc or zil. IT depends ™
P.s. did you get my thunderbolt reply to your email? Never did hear back but that might be okay.
First up, welcome to the forum, big fan of your channel!
I’m assuming you’ve read the ZFS docs – if not they’re worth a read to get an idea of what performance to expect vs. what disc configuration to select. Also to get your head around permissions being set via datasets and not necessarily via unix ACLs.
ZFS is very different to traditional RAID.
Essentially for VDEV configuration, it’s a trade-off:
number of VDEVs to stripe across vs.
level of redundancy per VDEV
If speed is a concern, definitely consider mirror VDEVs instead of RAIDZx. Mirrors can read different data from both (or all) drives in a mirror for better throughput unlike most other mirror implementations (because ZFS can verify the data by hash, rather than other copy). Also with mirrors you can fit more VDEVs into a given number of spindles, which means more IOPs as well.
One other big gotcha:
When creating your pool pay attention and make sure the “ashift” setting on your drives is 12 (4096 byte sectors); newer ZFS installs will probably detect this or default to 4096b sectors properly, but drives lie to the OS and if you end up with 512b sectors its a a case of
terrible performance
tears
pool rebuild
Also for VM usage, look for the dataset option for “ATIME” and set it to off.
Otherwise you’re generating writes for last access time, which will also tank performance on vm virtual disk files.
Can’t remove special device from pool once you add it. If you don’t have it mirrored, the special vdev can take out the pool.
Once special vdev fills up, new data goes into the main pool and never gets moved into the special vdev. You could end up with a special vdev with old data you almost never read.
My advice to OP:
Don’t bother with special vdev’s until ZFS supports removing them.
SLOG should be mirrored to prevent single point of failure. If a single SLOG device dies while the machine is running, ZFS can simply remove it and operate without it. If the power goes out and it dies and doesn’t come back on reboot… I guess I don’t have to tell you what happens to a database with a missing log file.
If you only have the single SSD, just use it for L2ARC.
@Dave_Plummer I saw your video on the windows storage spaces setup.
the reason you could only get 63TB out of the 203TB pool in a single volume is because the storage spaces gui is gimped. Using powershell to create a storage spaces volume will let you access the settings to go beyond the 63 (64)TB limit. Also the gui will only let you create a 3 column wide volume (assuming you choose a parity volume) which is not always what you want, this is analogous to only ever striping the data across 3 drives which has performance and usable space efficiency implications.