Abysmal TrueNAS performance

Hello. For the past several weeks, I’ve been witnessing truly terrible read/write performance on my TrueNAS home server, and I can not determine why. There are no warning signs on the GUI, and my 10G network card is working flawlessly. Basically, I determined it to be an internal issue as I can demonstrate with this screenshot:

TrueNAS

That is a single large file (33GB) being copied from one folder to another inside the NAS so there is no network involvement. The speed shoots up to 300/400 MB/s then drops back down to single digits, sometimes coming to a complete stop before shooting back up again. It’s as if it’s running out of cache or something.

Here are my TrueNAS specs:

TrueNAS-12.0-U8.1
Xeon E5-2670
SUPERMICRO X9SRA LGA 2011
Seagate Exos X18 Enterprise (ST18000NM000J) 18TB x 10
256GB (8 x 32GB) SK Hynix DDR3-1866 Registered ECC - HMT84GL7AMR4C-RD
Corsair RM650
TRENDnet 10 Gigabit TEG-10GECTX

Thanks folks.

PS: Running Z-2 Raid

Have you tried a transfer from you PC to your NAS and vice versa (using an SSD of NVMe drive in your PC)? Are the speeds faster when you do that?

Of course. The problem persists, same inconsistent transfer rates and coming to a halt in the middle of a transfer. Like I said, it’s not a network issue.

You can start by upgrading to the recommended/supported version which is 13.0-U1.1
Use gstat to see what’s going on regarding the HDDs from GEOMs point of view,
If you generate a long transfer using lets say iperf3, is the network performance consistent and if at what rates? You might also want to look at interrupts during transfers.

I would advise against 13. I had a few issues. 12 U8.1 is pretty stable right now

  • How are the hard drive connected? On board SATA or an HBA?

  • How are the 10 Drives configured? a big RAIDZ2?

  • Can you iperf at 10Gb/s?

  • How is CPU load during that?

  • Is that read and write? Or just write? Honestly I would not expect very good write performance with a big RAIDZ2 and no caching

Which doesn’t say anything at all unfortunately. FreeBSD 13.1 has a lot of improvements compared to 12.3 and while there might be I personally haven’t experienced any regressions on various of platforms.

OP’s not running cutting edge hardware here, there is nothing listed in his setup that should be having a problem in his current version

Based on? There are a number of improvements to the subsystem and kernel which would be very much beneficial. Just because you have aging hardware doesn’t mean that a newer software necessarily will be slower than the previous version. It’s worth mentioning that 12.X is going away so it makes little sense to troubleshoot an essentially dead platform.

@Hostile
Just for your information, iperf and iperf3 are not the same packages (not typos) :wink:

Does Windows file transfer copy files from network to PC to then write it back? that could explain the ~1/2 bandwidth.

Pool at high capacity (>60%) with serious fragmentation? ARC should mitigate those problems, but usually pool performance degrades quite considerably at higher %

might be sync or ARC related…not sure

It’s perhaps a bit more logical to look at the basics first and then look at what potentially ZFS is doing just for the sake of making sure the hardware performs as expected.

Please post your complete zpool topology.

zpool status

3 Likes

Based on there is no widespread reports of U8.1 having performance issues…

I take that as you don’t have any arguments against upgrading since the current version is on the chopping block. Rule #1 is to have software that’s up to date and supported, there’s no point in troubleshooting a legacy version which the vendor have little no to interest in for no obvious reason.

Yes, as I said, I do. Anyway, I’m not getting into some silly argument with you man, we’re hijackings this guys thread, peace out man

Guys I’m sorry it took me a while to reply but we had a local power outage and I couldn’t do anything while that was happening.

Thank you for all your suggestions and recommendations.

I did run iperf and confirmed everything to be working correctly AFTER I changed the Cat7 cable that was connected to my NAS as I noticed some fraying around the connector. I’m getting a solid 5.05 Gbits/sec indicating a 5G link. I don’t know what happened to my other 5Gs but I’ll take it for now.

Speaking of, do you guys have a recommendation for a solid non-amazon basic Cat7 Cable that can guarantee 10G?

Cat7 isn’t a real TIA standard, so almost every one is pure garbage. Just buy any good quality Cat6a cable

I’m a big fan of the Monoprice Slimrun Cat6a cables

2 Likes

Monoprice even has some cat8 cable, why not go big?

Because it’s probably crap

Cat6a is fine for 10G. Higher rating for patch cables isn’t necessarily better. More bend radius restrictions, less flexible cables etc.

If you’re crimping your own cables my advice would be don’t. Trying to crimp cat 7 yourself… it isn’t really meant for patch cables. It’s made for structured in building cabling runs.

Equip works well in my experience
https://www.equip-info.net/equipinfo_en/equip/cables-adapters/cables/cables/network-cables/cat-6a

There’s also CAT 8.1/8.2 but that’s kinda useless for your use-case unless it costs the same as 6a. If you’re getting installation cable make sure the manufacturer actually provides a datasheet for the cable rather than just a claim.

1 Like