Odd TrueNAS Scale NFS issues

Operating Systems & Open Source Linux
, ,
1

Update:

Issue suddenly reappeared without the Optane, so that has been ruled out as the issue. Performance is also vastly improved using a different NFS Server, see further below in the thread.

TL;DR:

Recently, I diagnosed @Bumperdoo’s System’s Performance editing Video from an NFS-attached TrueNAS Scale System. Performance had been rock-solid, but suddenly became stuttery and there was general instability on the System.

After trying different Kernels and Drivers, RHEL Family Distributions, as well as much NFS troubleshooting with nfs(io)stat and the likes…the Problem was solved by removing a Chipset-via-U.2-connected Optane P5800X and repurposing a secondary Samsung 990 Pro for the Boot Drive.

Now I’m trying to wrap my head around how the Optane could have been the cause for the issue.
This post is mainly for the purpose of curiostiy about the odd solution.

Relevant Hardware:

Workstation:

CPU: AMD Threadripper Pro 5975WX
Motherboard: Supermicro M12SWA-TF
RAM: 8x16 GB Micron DDR4 ECC RDIMM (tested fine wth memtest86+)
NIC in CPU PCIe Slot: Intel XXV710 Dual 25Gbit
GPUs, also in CPU PCIe Slots: Dual RTX 4090
I/O: Blackmagic Decklink Mini SDI Card to attach a reference monitor for Color work
Storage: Optane P5800X and Samsung 990 Pro as a secondary drive. (originally)

OS: Fedora 38 / Rocky Linux 9 (issue occurred under both)

Everything has plenty cooling and Performance of the individual components has been verified.

Server:

Amd Epyc / Supermicro H12-based Custom-built Server with 256GB of DDR4 ECC RDIMMs and another Intel XXV710. Running TrueNAS Scale with a large RAIDZ-2 WD Gold Array and a smaller Flash-based Pool. NFS Set to Standard (Syncing in this use case), no atime for the media datasets.
I won’t go into too much detail here since the Server has been performing fine and continues to do so.

Problem Description and Troubleshooting:

After over half a year of great performance Editing and Rendering Video from the above NAS in Davinci Resolve on above Workstation, Performance suddenly became stuttery, Video Playback and scrubbing was not smooth anymore.

Sometimes Performance looked fine for a short amount of time when starting a fresh workload, but would then quickly degrade again.

A few of the troubleshooting steps attempted:

  • Trying different Kernels and Nvidia Drivers
  • Making sure Network Performance was as expected (24.7Gbit/s between Client and Server both ways) and that there was no problem caused by the NIC under heavy load or with different packet sizes, all was OK, no packet drop, bad latency or retransmissions
  • Changin NFS rsize and wsize, packet buffers on the OS Level, etc.
  • Synthetic Sequential Loads were fine, but file transfers did show the same problem.
  • Caching on both the Client and Server Side was behaving normally, as before and expected.
  • Switching between Fedora 38 and Rocky Linux 9
  • Using nfstat and nfsiostat to closely watch all the great metrics they provide, all was fine for one exception of reported throughput in nfsiostat
  • Comparing Performance to another NFS Client - no issues on that one
  • Checking whether GPUs or Decklink Card were reporting problems or creating the bottleneck, both not the case.

None of the above pointed at anything being wrong or showed signs of unusual behavior.

Except for nfsiostat reporting less-than-expected throughput in KB/s - not the amount of Video Davinci was actually going through, but I could not find any further reason for that after investingating Cache and NIC Behavior / Metrics.

Odd Solution / Open Question:

Removing the P5800X and using the 990 Pro as the Boot Drive instead solved the issue instantly. There was no change in configuration on either the Hardware or Software Side.

The only noticeable change in metrics was a correctly reported throughput in nfsiostat.

Now I am left wondering: How could the P5800X, which performs fine in Benchmarks and reports as healthy in its SMART Data - have been the cause of the problem?

It was connected via the Chipset, because the U.2 Connector on the Board is routed that way - maybe that created an underlying issue I wasn’t able to diagnose easily? The Samung is on CPU PCIe Lanes instead.

Happy to hear any ideas.

1 Like
2

So I understand correctly, the P5800X was connected to the chipset while the issue was occurring?
I’m no low-level hardware expert, and am going to speculate.
Maybe the chipset was being overloaded with I/O from the P5800X, with its low latency it can hammer I/O more than other SSDs. Perhaps some kind of resource contention on the chipset controller, as other devices on the same chipset lanes wouldn’t have enough multiplexed bandwidth, such as networking. Load-induced latency, a race condition?
But that doesn’t explain how it was working fine before. We’re missing something (or somethings).

Try the P5800X on the CPU lanes, and the Samsung on the chipset.
Longshot, but also try updating the firmware on the P5800X, one came out a few months ago, they still publish firmware updates.

2 Likes
3

I consider this to be a logical conclusion, but all the other PCIe AICs including the Network Card are connected to CPU Lanes. Therefore I wasn’t able to pin down how a contention on the Chipset could have caused the behavior.

We weren’t able to do it in the same test setup, since all CPU PCIe Slots are either used or blocked and we don’t have an M.2 → U.2 Adapter on hand. Also, the U.2 Connector provides the only exposed PCIe Lanes from the Chipset.

While it might not directly explain the change that happened, it’s worth a try!
Looks like there was a new firmware released in December of 2022.

4

We verified that the Optane was on the newest Firmware from December already.

- 0 Intel Optane SSD P5800X Series -

Bootloader : LB3B0559
Capacity : 372.61 GB (400,088,457,216 bytes)
DevicePath : /dev/nvme0n1
DeviceStatus : Healthy
Firmware : L0310600
FirmwareUpdateAvailable : The selected drive contains current firmware as of this tool release.
Index : 0
MaximumLBA : 781422767
ModelNumber : INTEL SSDPF21Q400GB
NamespaceId : 1
ProductFamily : Intel Optane SSD P5800X Series
SMARTEnabled : True
SectorDataSize : 512
SerialNumber : Redacted
1 Like
5

sigh

This one has us truly perplexed. Seems the problem is not related to the Optane after more testing.

NFS on Windows to NFS on the TrueNAS Scale works flawlessly.
NFS on Rocky Linux to NFS on TrueNAS Scale has network issues.
NFS on Rocky Linux to SAMBA on TrueNAS Scale is fine.

Truly puzzling. Investingation continues to resolve the root problem.

Any ideas?

6

Have you done any NFS tuning? What are the mount options on the linux machines and what does your export look like? What version of NFS are you using amd are the client systems using the same version or are the using a lower version?

1 Like
7

Tried both NFSv3 and v4 on Client and Server the same, different r/wsize options, sync and async, packet buffers on the NICs, etc.

For now I’m assuming the problem lies on the TrueNAS Machine because I can’t reproduce the issue with a different NFS Server or even on my own Server running the same version of Debian and OpenZFS.

We’ll completely reinstall / reconfigure the TrueNAS Host and report back.

8

You may have encountered a bug with “TrueOS”. I know that there are some new features in NFS v4.1 that do not exist 0n 4.0 when dealing with security. The fallbacl to v3 should just work in most cases unless your security settings are restrictive or prevent down grading.

Another thing to check are the settings on TrueNAS to ser how many threads are being reserved for NFS transfers. By default, the thread count is pretry low and BSD is known to use extremely conservative defaults. I cannot rember the serting right now as I am on travel and away from my NAS. I am using Debian Testing for my NAS (Raspberry Pi 4 with the Radxa SATA hat) after running into issues with Ubuntu and RPC Bind doing some weird things in regard to NFS.

9

This isn’t a very useful problem statement. What benchmark numbers show a problem, and what numbers did you previously have, or would you expect them to be?

Try local benchmarks on the server versus on the workstation across NFS, versus a different NFS server. Maybe you’ll narrow down the location of the problem just a bit.

10

We have that data and can post it for clarity.

Yah, we did all that. Basic NFS share on a Rocky Linux install to the workstation was fine. TrueNAS Scale to the same workstation resulted in poor performance.

In the end, we concluded that TrueNAS Scale’s NFS shares are somehow flawed; didn’t believe it at first but after a lot of testing it’s our natural conclusion.

11

Same behavior on NFS v4 and v3.

It was set to 32 by default but I tried both much lower and higher values for the purpose of troubleshooting. No improvement there.

Just to clarify, we’re on TrueNAS Scale (based on Debian).
Which makes the performance issues so odd.

Testing with the NVME based Pool, to rule out Spinning Rust being the Bottleneck:
Local Performance on the TN Scale Host is over 1GB/s in any tested Scenario.
Synchronous Speeds for Read and Write tested with fio and different block / file sizes are well over 1.2GB/s.
Asynchronous Speeds for Read and Write tested with fio and different block / file sizes are over 2.3 GB/s.

Sequential synchronous Writes using the real workload of video files from a Host connected via Ethernet with a 24.7 Gbit/s iperf3 Throughput, each file being 2GB in size and the test project 300GB Total:

Less than 300 MB/s in both read and Write via NFS, double that via Samba, both under Rocky Linux.
Little over 300 MB/s via NFS when setting the Dataset to force asynchronous Writes (only for testing purposes), but still not stable enough to not get stuttery Video when editing.
Same Transfers under Windows were over 1GB/s and we used to get similar Performance via NFS under Linux as well.

Expected Performance, as tested on my own Debian System with much worse Hardware running an NVME Pool with similar local Performance is 1 GB/s via NFS, synchronous, for large groups of 2GB Video Chunks.

As stated by bumperdoo, the problem is not present with a different NFS Server or locally.

12

That number should be at least a multiple of the core count that you have in your CPU. I am running on an RPi4 and I have my thread count set to 32. That is the sweet spot for transfers from the server to the client. Any less and I cannot saturate the ethernet from the pie. Any more and I get diminishing returns. The flip side is that my write speeds are pretty low, like 2 - 10 MiB per second versus my read reads are like 58MiB per second. By The NAS is meant to be a backup server that primarily streams media to the phones and TVs in the house.

Well darn. I am a Debian shill, so I may be able to help then. I that TrueNAS dropped all other derivatives to only support TrueOS.

Question: What firewall comes with TrueNAS Scale. Are they using ufw? IPTables? IT may be the firewall that is causing slowdown for you if it is not the NFS settings. Also check to see if apparmour is running. I would say turn off any firewall or AppArmour or SELinux and see if that makes a difference. If so, then you will need to create some rules in those applications to allow your traffic to pass unobstructed.

I am sorry if this all seems random. These are things that you need to do on Debian, but I have no experience with TrueNAS.

1 Like
13

Sorry about the late reply, thought I sent one off but must’ve forgot to press reply.

We had tested different NFS Thread Counts before, but I just retested to make sure I remembered correctly - setting them to 64, 128, 256 makes almost no difference. Still less than 300 MB/s sustained.

I know what you’re talking about though - I usually also see a performance jump going to 4x-8x the number of CPU counts (even in VMs!) on my machines.

These are all good ideas, thanks!
Problem is that TrueNAS Scale is architected as an Appliance Distro so anything that’s not configured through the TrueNAS Configuration Tools is very likely to break on Upgrade of the System or even on restart.
And since this system is not running in a homelab environment I have to consider running configuration that is likely to fail during future Updates or even in normal operation to be a bad idea.
But I’ll think about it a bit more, maybe I can find a way around that.

1 Like
14

This topic was automatically closed 273 days after the last reply. New replies are no longer allowed.