TrueNAS Performance Questions

Hi All,

I recently started foolin around with an old server in my lab. I’m looking to see if my performance numbers are out-of-whack because they seem low to me. I’ve built a poor-man’s all flash array using the following

  • A Supermicro 216 Chassis
  • A Supermicro x8dth-6 motherboard (SAS2008-IT onboard)
  • 2 additional LSI 9211-8is
  • 2x Xeon X5680s
  • 96GB of RAM
  • 22 Crucial BX500 120GB SSDs (11 mirrored VDEVs) ashift=12
  • Intel X520-DA2
  • Running TrueNAS Scale latest beta

fio results:

Sooo 16,000 IOPS and 2000 MiB/s read and write…

A ZVOL Mounted via ISCSI on Windows over a 10gigabit network:


As a point of reference I compared these results to my production server. The production server is faster than the all flash server? This is NOT apples-to-apples but I’m rather confused??

  • Dell R720
  • H310 IT Mode
  • LSI 9205-8i IT Mode
  • 2x Xeon E5-2920s
  • 192GB RAM
  • Fusion Pool with 12x4TB WD RED (CMR) drives and 2x Samsung SM953 480GB NVME SSDs
  • TrueNAS CORE 12.0U6

Production server FIO results

Sooo 17,000 IOPS and 2100 MiB/s read and write…

A SMB share over the same 10gigabit network to the same desktop


I’m going to try and re-do the testing with some 9205-IT mode cards, but since my system is PCI-E 2.0 I’m not sure if it will have an appreciable benefit. SAS2308 is generally better than SAS2008 chips in any case. But still, I was expecting closer to 3500 MB/s…at least inline with modern M.2 performance…with 3 separate 2008 controllers and 24 drives with no SAS expander I’m not sure why that’s not possible??

I can also try and swap the drives for some old 180GB Intel SSD 520s I have to see if that changes anything. The CT120BX500SSD1 SSDs I have are DRAM-less and any efficiency increase with more modern NAND vs the old 520s is probably lost to that. Worth experimenting, I wish I had a 3rd set of drives to compare against but that’s all I have to play with.

I’ve also bid on an X9DRi-LN4F so I can get some faster Ivy Bridge Xeons (E5-2667 v2?) and see if that helps after I do the above testing…

Any other thoughts are appreciated, I know this is all older hardware but I’m not sure what the biggest bottleneck is…This should be significantly faster than 12 spinning disks in a Raid-Z2 by the nature of the fact that they in a mirrors pool and they are SSDs…?

So I did the SSD swapout to the SSD Intel 520s

Now we are at 12K IOPS and 1500 MB/s read and write.

I know these are really old Sandforce drives, so I wasn’t sure what to expect here. Still, performance seems rather low…

Crystal Disk Mark over ISCSI performance

I will try and swap in the LSI 9205s tomorrow. Any other ideas are appreciated.

Additional testing has ensued…

Replaced 9211s with 9205s

13k IOPS and 1700MB/s

Less than with the Crucial drives, faster than 9211s.

Won the action for the LGA-2011 board. Will test with the E5-2620s that it came with (the same CPUs that are in my production server).

RAM is the same exact DIMMS as are in my production server. Experiment is on hold until that comes. If it’s not significantly faster with the new board I’m sorta at a loss??

Is my testing methodology not correct?

This is not my territory, so I may be totally barking up the wrong tree, but… I see your “slow” system is using many SSDs. Are you aware of Wendell’s video and post re slow I/O performance with many NVME SSDs?

To be sure, the circumstances were different, but as I understood it the root cause had to do with getting so many I/O interrupts so fast that the Linux kernel tripped over its own feet. Wendell says the Linux kernel has since been fixed, but something similar might be affecting your BSD system.

Just a possibility to consider. Good luck.

Hey man,
Those types of issues shouldn’t crop up on SATA SSDs, that level of insanity needs faster NVME drives for the kernel to be a problem.

So my new motherboard arrived today…and as I had hoped… It is significantly faster…

fio --bs=128k --direct=1 --directory=/mnt/lol/fio --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=12 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based


Sitting at 25.k IOPS and 3215MB/s

I wish I had some SAS3 cards/SAS3 backplane to see if performance scales JUST by moving to SAS3 hardware…but they are still pretty pricey

I think I will pickup some faster single threaded CPUs next and see what benefit that gives me. 2620s don’t exactly scream, and the X5680s I had in the other board are probably faster. It’s kinda silly, because over ISCSI it’s still only a little faster than a single SATA SSD…


Congratulations on the doubling of performance! Thanks for sharing your experience.

Received new CPUs already… Performance hasn’t changed?



26K IOPS 3272MB/s

To be clear, it does seem to be scaling linearly, so maybe this is as fast as these drives go?

This is a single drive:

With 11 VDEVs of 2-drive mirrors I am getting 10x the performance

I’ve also tested with all 22 drives in a single RAIDZ1

23k IOPS and 2800MB/s

Performance is pretty close to mirrors, which goes against conventional wisdom I’ve seen here. It was always my understanding that mirrors had a high cost in storage efficiency but were always faster than RAIDZ. Although, we are talking about 22 drives and only a single drive worth of parity…not exactly production ready.

Two Raid-Z1 VDEVs of 11 drives each yields the best performance/storage efficiency so far

28k IOPS and 3500MB/s.

Still, not sure if I would put into production 1 drive of parity with 11 disks.

So next I made a 20-drive ZPool with 4 5-drive vdevs in a raid-z1

Losing 2 drives as “hot spares” plus 3 for parity for a total of 5 drives is still better storage efficiency than the mirrors(which lost 11), and still faster.

26K IOPS and 3200MB/s

Finally, I made an 11-drive-per-vdev RAIDZ-2 of 2 VDEVs. This offers better storage efficiency than the 3 RAIDZ-1 VDEV arrangement.

26k IOPS and 3200MB/s still…which is still on par with the mirror array and from my test appears to be more consistent

So some takeaways…

  • I’m not going to break 30k IOPS anytime soon
  • SAS3 cards may be an answer, but I don’t honestly think they will be
  • I have 5 PCIE 16x slots I can put NVME drives and I’m fairly certain that I can do PCI-E bifurcation in this board so I may be able to do 10xNVME drives on this platform to see what that does.
  • If anyone has sone 256gb nvme drives from old laptops or something and you want to donate to the cause for science I’m here xD

I hope this helps someone later…

1 Like

Conventional wisdom also wouldn’t put 22 drives in a z1 :slight_smile: How is the resilver time with that config? I wouldn’t like to do this on HDDs, but SSDs should still be fine although vastly higher than mirror resilver. But I’ve seen several benchmarks with HDDs and SSDs saying that Raidz1 performance is pretty much mirror speed as long as you can throw dozens of drives into the pool.

And yeah, I love those walls of fio output. I just don’t have the drive numbers to make extensive testing of differing configs myself.

1 Like

The conventional wisdom is that mirrors will always bet faster than raidz, or at least that’s what I’ve always believed.

To be clear, I wouldn’t ever put 22 drives in a z1 for more than FIO testing xD

I think the general rule is to start breaking things up into separate pools past 8 or 10 drives.

Not separate pools, but separate VDEVs.

In my testing of 24 drives (obviously more drives are needed to test this), 3 smaller VDEV of 5 drives each in RAID Z-1 are not faster or more storage efficient than two 11-drive ZVOLS. I think it’s safe to be + or - 20% of 10 drives per VDEV (between 8 and 12) depending on specific situations.

1 Like

Right, thanks. I haven’t played with ZFS enough yet to fully grok the terminology.

For giggles, I wanted to get every drop of performance out of this. I put all 22 drives in a single vdev striped

38K IOPS 4700MB/s

1 Like

Join the Church! It’s awesome. Before I built my new server, I was playing around with a fist full of old USB flash drives, building (and destroying) my first pool.

@NicKF how about a 22-way mirror? For Science!

1 Like

I’ve had a TrueNAS box as part of the homelab for a while now, but haven’t had the time to really dig in yet.

For science!

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.