@wardtj Sorry if this is a stupid question, but would these tweaks apply to SAS2/3 zvols as well ?
If not, what will ?
I recently enabled RDMA on my mikrotik switch (thanks MT !!) and i’m exploring this world of extremely fast networking (still only 25 gbe) in my homelab environment and was absolutely shocked at the speed my SSD nas was able to serve. Currently I’m limited by the speed the ssd’s in my computer and my CPU.
I did also notice the EPYC 7302p in my truenas server had pretty high CPU utilization whenever I ran any Crystal Diskmark tests. The VM was connected to an iSCSI share on Truenas via dual 25gbe paths.
Any insight on where I can research any more tweaks, would be greatly appreciated.
I was looking at the overall balance. This set up regularly supports 1M iops in vsphere 8. I wanted more space for the investment in drives. I use tiered pools as well. NVMe for high iops, SATA SSD for bulk data, backup and non-performance workloads, and SATA HDD for low performance DR copies of production and backups.
Yes these apply to SAS drives , but would avoid HDD with these settings as its really easy to overrun the drives and cause large I/O backlogs.
You should see high CPU usage under load like crystal. At 25g you are moving 2.5GB/s which will drive I/O threads. If you have slower drives I/O wait starts creeping up driving CPU time up. ZFS will not cache in ARC large sequential writes which crystal will do BTW.