High latency with Debian VM on TrueNAS Scale

Operating Systems & Open Source Virtualization
1

Hi folks,

I hope you guys can give me a north position. I’m having a latency issue on my Debian VM that I have running on top of my TrueNAS Scale homelab.

I have two machines running TrueNAS Scale 25.10.5, both of them have a single Debian VM that runs my docker workflows.

The problematic host is a Ryzen 5700G with 128GB of RAM, it has a RTX 5060 GPU thats used for compute + decoded and that GPU is passed through to the VM. On the MB 4 SATA ports there are 4 NAS 8TB HDDs divided onto two pools data0 and data1, both are two disk 8TB mirror.

I also have two Samsung 500GB SSDs on the two NVMe ports, those are on a pool called fast1

The Debian VM has 7 pinned cores of the 8 available (with their respective threads) and it has 60GB of dedicated RAM. The other ~60GB are dedicated mostly to ARC with a small percentage to the operation of the TrueNAS host which does snapshotting and a smb host pretty much.

On the TrueNAS host I have zero latency issues, everything is extremely fast, every test i throw at it it just works perfectly.

On the Debian VM if I try to for instance do a sudo docker pull ollama/ollama which is a 3GB image the system “hangs”, what do I mean by hanging is, its still responsive and any open SSH sessions are still responsive, but if i try for instance login into a new SSH session i got hung until the extraction of the image finishes, but its not limited to SSH, any trivial command like a apt search something starts to get hangs of about 10 seconds and sometimes it only finishes the command after the extraction process finishes and anything IO demanding in the system seems to get those hangs. If I try to do a htop or a top there’s a total of zero high demand on the hardware for both the VM or the host, they are breezing awat at their single digit percentage on both RAM and CPU, iostat -dx 1` is weirdly zeroed, like, when docker start the extraction instead of increasing the latency it actually starts to zero it. on cat /proc/meminfo Dirty normally hovers around 100MB, but when it does some of those complex actions, and extracting ollama is just one of those, anything random IO demanding seems to trigger it, the Dirty memory gets to hundreds of bytes. Writeback is 0KB all the time pretty much. I’ve also zpool iostat -v 1 on the host while the VM is hung and there’s no obvious spike of latency and the transfer rate is about 80MB/s.

hdparm -Tt shows about 1.3GB/s on the NVMe SSDs and at minimum 215MB/s on the HDDs, so no sequential issues. Smart shows zero issues similar to zpool status, no scrub issues.

Load average is 1.99 for the host and 0.47 for the VM. The VM is using 7.3GB of 63GB and the host is 12.2GB of 128GB. Those stats are from htop and free -m

If someone has any idea of what may be causing this issue, be any VM setting, the hypervisor, truenas, zfs, hardware, for now I have no clue of what may be causing this issue bc nothing is reporting a problem and I cannot reproduce with synthetics.

2

Those pauses are on network activity and you’ve not talked about network setup. What’s the configuration and what troubleshooting have you done?

3

I encountered a very similar problem, consulted the AI oracles and they correctly suggested trying to disable sync on the dataset temporarily. And sure enough that magically “fixed” it. Apparently ZFS can detect that the drives don’t have PLP and thus go with a conservative sync approach.

You should be able to check with fio…the test results came back comically slow for me so you can actually measure it while troubleshooting

Apparently adding a SLOG with PLP could fix it more correctly than disabling sync, but in my case its a single drive so probably just switching to ext4 and calling it a day

4

That was that,

It probably tried to sync every operation, if we understand the risks of running that zvol unsynched it does actually make the OS usable. You dont need to do it for every dataset you have attached to that VM, just the rootfs is enough.

5

I don’t think this is the case. ZFS is sending a regular disk flush, irrespective of PLP or non-PLP. It’s the SSD that’s different. PLP SSDs can return flush fast, non-PLP SSDs need to mull it over until it’s definitely written (Or not, depending on the SSD ¯\_(ツ)_/¯).

SLOG isn’t really going to do much for NVME SSDs unless you have a really good one.

In this case I’d maybe update the firmware for your SSDs. There may be a bug making flushes, etc, take longer. It could also be due to no TRIM - maybe try enabling zfs-trim-weekly systemd timer?

1 Like
7

Im not an expert at all,

In my case what I could investigate is that since there are more layers ext4 → kernel → VirtIO → ZFS → Kernel it made the flush chain far more slower than it normally is, so my non-PLP SSD cannot handle the amount of flushes its being submitted once a single flush is requested high up in the VM. Thats looks like the reason the problem only happens on the VM and on TrueNAS itself the performance is quite okay.

That what I understoood after asking that exact same question to AI, i made some new tests with fio on both the host and the VM and the latency on the VM is far higher than on the host directly.

Anyways. I got a suggestion to actually migrate my workflow directly on TrueNAS instead of using a VM and honestly? Thats a good idea, my workflow is 100% containerized and there’s no reason I cant run on a stock TrueNAS setup and then those SSDs will be retired since their only job was to hold the VM rootfs.

1 Like