OpenZFS 2.3, Direct IO, and NVMe

Hardware Hub Storage
, ,
1

The 2.3.0 release of OpenZFS adds support for direct IO, which is briefly described in the release notes:

More, from the merged pull request:

"By adding Direct IO support to ZFS, the ARC can be bypassed when issuing reads/writes. There are certain cases where caching data in the ARC can decrease overall performance. In particular the performance of ZPool’s composed of NVMe devices displayed poor read/write performance due to the extra overhead of memcpy’s issued to the ARC.

There are also cases where caching in the ARC may not make sense such as when data will not be referenced later. By using the O_DIRECT flag, unnecessary data copies to the ARC can be avoided."

And then described in greater detail in this talk from a few years ago:

It sounds compelling to me. Has anyone had time to experiment with a direct IO-enabled pool? (Is it yet too new?)

  • What are the performance characteristics of direct IO in practice?
  • How does it / does it not improve performance of NVMe-only ZFS topologies?
  • Is it a situational optimization or is it more generally helpful?

Thanks.

1 Like
2

I’ve been using direct=standard for awhile now and didn’t really notice any difference, however out of pure morbid curiosity I switched to direct=always today.

dd if=/dev/zero of=/mnt/zero bs=4M
530669+0 records in
530669+0 records out
2225787109376 bytes (2.2 TB, 2.0 TiB) copied, 250.458 s, 8.9 GB/s

Definitely see a difference there. Almost an extra 4 GB/s on sequential writes.

1 Like