Hi
I have and HP 2CPU server with 2x8 cores (Intel(R) Xeon(R) Silver 4110 CPU @ 2.10GHz) with around 377GB RAM running TrueNAS Scale
I currently have a HP 19" M6720 Storage 24x 3,5 storage shelf running SATA disks, 24x Toshiba 16TB.
Everything is working well but I expect to need to expand on storage in about half a year and I am planning to buy a second shelf.
My first question is, should I connect the second shelf to the first shelf or to the HBA? What is the best both for stability and performance?
I can see that I can add cables to both heads on my existing storage shelf but I don’t see any usage for that now that I am running with only SATA disks, what is your opinion here?
I am running with 3x ZFS RAIDZ2 with 8 disks per vdev.
Four SAMSUNG MZ1LB1T9HALS-00007, with two as chache and two as metadata.
I also have 2x Intel® Optane™ SSD DC P4801X 100GB disks for log.
An extra question, can I see if I have enough storage for Metadata and cache?
I primarly run NFS for and mount virtual machines on a Proxmox server, but I also run NFS for my Kubernetes storage.
I use a ConnecX-5 dual 40GE connection and all my physical servers use 40GE.
I would connect directly to the HBA. This way, the disks are accessible in case of a failure in the other storage shelf. Not sure how much good that would do in the bigger picture (I assume the zfs pool would still suffer), but that’s my reasoning.
Not sure how much concurrent traffic your zfs pool gets. But my hunch is that given the number of spindles your pool performance is limited by the log disk configuration. The Optanes have plenty of bandwidth, but not sufficient when loaded with low queue depth, small recordsize workload.
In my (similar) setup I find that I can double write performance to my pool by using mirrors of partitioned log devices (still two physical devices and always mirroring partitions of different physical devices). This setup should not increase risk of failure, but force zfs to use a higher queue depth workload onto the devices.
Maybe there is a way to configure it using zfs parameters, but I have not yet found it.
Same thought as above probably applies to these vdevs.
For expanding your storage setup with a second shelf in your TrueNAS Scale system, it’s generally recommended to connect the new shelf to the existing one rather than directly to the HBA on the server. This approach provides better management and scalability, allowing you to manage all storage devices and disks more efficiently within TrueNAS. It also simplifies cabling.
Do you think 50GB is enough? I should be able to use nvme-cli to split up the disks in two namespaces/partitions per disk and then use those four as a big mirror with four devices, or should I have two mirrors?
you need maximum throughput times the txg_timeout in capacity on the SLOG. txg timeout is 5 seconds by default. So 50GB will be plenty for anything up to 10GB/s of writes (which is a lot)
two mirrors. Just add a mirror with two disks and then add the other disks as a mirror. You can add two disks as long as you have disks. Data gets automatically distributed between the mirrors and results in double performance more or less.
