Sequential Write Performance Issue on ZFS 2.3.x vs 2.2.x using LZ4 Compression on NVMe Drives

I’ve got a problem with sequential write performance on ZFS 2.3.x vs 2.2.x.
OS: Debian 13.1
NVME: 4 x Sandisk 2TB M.2 PCIe Gen5 NVMe Black SN8100
CPU: AMD EPYC 4564P

I’m testing by compiling ZFS from source, installing the generated *.deb files, and rebooting to make sure the kernel loads the new modules.
Every time I set these parameters:

echo "4294967295" > /sys/module/zfs/parameters/zfs_arc_min
echo "4294967296" > /sys/module/zfs/parameters/zfs_arc_max

Then I create the pool with:

zpool create -f \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl -O xattr=sa -O dnodesize=auto \
-O compression=lz4 \
-O utf8only=off -O normalization=none \
-O relatime=on \
-O canmount=on -O mountpoint=/ztest -O snapdir=visible \
ztest mirror \
/dev/nvme0n1 \
/dev/nvme1n1 \
mirror \
/dev/nvme2n1 \
/dev/nvme3n1

Testing seq. read/write with fio:

fio --filename=/ztest/test.bin --direct=1 --rw=read --ioengine=io_uring --bs=2m --iodepth=64 --size=10G --numjobs=1 --runtime=60 --time_base=1 --group_reporting --name=test-seq-read --

fio --filename=/ztest/test.bin --direct=1 --rw=write --ioengine=io_uring --bs=2m --iodepth=64 --size=10G --numjobs=1 --runtime=60 --time_base=1 --group_reporting --name=test-seq-write --

Sequential read performance shows a big improvement:

zfs v2.3.4:

fio-3.39
Starting 1 process
test-seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=19.5GiB/s][r=9962 IOPS][eta 00m:00s]
test-seq-read: (groupid=0, jobs=1): err= 0: pid=229665: Thu Nov 13 13:37:59 2025
read: IOPS=9966, BW=19.5GiB/s (20.9GB/s)(1168GiB/60006msec)
slat (nsec): min=331, max=3970.8k, avg=3094.65, stdev=17943.07
clat (usec): min=92, max=24643, avg=6416.33, stdev=644.56
lat (usec): min=281, max=24657, avg=6419.43, stdev=644.38
clat percentiles (usec):
| 1.00th=[ 4424], 5.00th=[ 5604], 10.00th=[ 5932], 20.00th=[ 6194],
| 30.00th=[ 6325], 40.00th=[ 6390], 50.00th=[ 6456], 60.00th=[ 6521],
| 70.00th=[ 6587], 80.00th=[ 6652], 90.00th=[ 6783], 95.00th=[ 6915],
| 99.00th=[ 8356], 99.50th=[ 9503], 99.90th=, 99.95th=,
| 99.99th=
bw ( MiB/s): min=19144, max=20160, per=100.00%, avg=19943.02, stdev=84.88, samples=119
iops : min= 9572, max=10080, avg=9971.50, stdev=42.43, samples=119
lat (usec) : 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.06%, 4=0.63%, 10=98.93%, 20=0.37%, 50=0.01%
cpu : usr=1.57%, sys=6.86%, ctx=552160, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=598043,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=19.5GiB/s (20.9GB/s), 19.5GiB/s-19.5GiB/s (20.9GB/s-20.9GB/s), io=1168GiB (1254GB), run=60006-60006msec

zfs v2.2.8:

fio-3.39
Starting 1 process
test-seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=5016MiB/s][r=2507 IOPS][eta 00m:00s]
test-seq-read: (groupid=0, jobs=1): err= 0: pid=102593: Sat Nov 15 00:37:30 2025
read: IOPS=2512, BW=5026MiB/s (5270MB/s)(295GiB/60013msec)
slat (nsec): min=330, max=11393k, avg=22868.39, stdev=197445.05
clat (usec): min=220, max=167763, avg=25434.27, stdev=13153.14
lat (usec): min=519, max=167768, avg=25457.14, stdev=13150.89
clat percentiles (msec):
| 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15],
| 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26],
| 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 43], 95.00th=[ 52],
| 99.00th=[ 70], 99.50th=[ 78], 99.90th=[ 94], 99.95th=[ 102],
| 99.99th=[ 121]
bw ( MiB/s): min= 4892, max= 5137, per=100.00%, avg=5029.29, stdev=50.95, samples=119
iops : min= 2446, max= 2568, avg=2514.25, stdev=25.32, samples=119
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=4.47%, 20=36.30%, 50=53.59%
lat (msec) : 100=5.57%, 250=0.06%
cpu : usr=2.96%, sys=4.71%, ctx=61753, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=150799,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
READ: bw=5026MiB/s (5270MB/s), 5026MiB/s-5026MiB/s (5270MB/s-5270MB/s), io=295GiB (316GB), run=60013-60013msec

But something got seriously broken with sequential write:

zfs v2.3.4:

fio-3.39
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=936MiB/s][w=468 IOPS][eta 00m:00s]
test-seq-write: (groupid=0, jobs=1): err= 0: pid=229787: Thu Nov 13 13:39:44 2025
write: IOPS=469, BW=939MiB/s (984MB/s)(55.1GiB/60135msec); 0 zone resets
slat (nsec): min=3386, max=98575, avg=25828.94, stdev=8272.96
clat (msec): min=3, max=266, avg=136.32, stdev= 8.07
lat (msec): min=3, max=266, avg=136.35, stdev= 8.07
clat percentiles (msec):
| 1.00th=[ 112], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136],
| 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 138], 60.00th=[ 138],
| 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 140],
| 99.00th=[ 144], 99.50th=[ 178], 99.90th=[ 209], 99.95th=[ 236],
| 99.99th=[ 264]
bw ( KiB/s): min=819200, max=1257472, per=100.00%, avg=961228.80, stdev=34982.79, samples=120
iops : min= 400, max= 614, avg=469.35, stdev=17.08, samples=120
lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.08%, 100=0.39%
lat (msec) : 250=99.46%, 500=0.03%
cpu : usr=1.27%, sys=0.08%, ctx=28254, majf=0, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,28224,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=939MiB/s (984MB/s), 939MiB/s-939MiB/s (984MB/s-984MB/s), io=55.1GiB (59.2GB), run=60135-60135msec

zfs v2.2.8:

fio-3.39
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=3341MiB/s][w=1670 IOPS][eta 00m:00s]
test-seq-write: (groupid=0, jobs=1): err= 0: pid=102725: Sat Nov 15 00:39:46 2025
write: IOPS=1749, BW=3500MiB/s (3670MB/s)(205GiB/60048msec); 0 zone resets
slat (usec): min=4, max=56978, avg=109.01, stdev=564.70
clat (msec): min=4, max=220, avg=36.45, stdev=35.00
lat (msec): min=4, max=220, avg=36.56, stdev=35.07
clat percentiles (msec):
| 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21],
| 70.00th=[ 29], 80.00th=[ 56], 90.00th=[ 99], 95.00th=[ 126],
| 99.00th=[ 138], 99.50th=[ 142], 99.90th=[ 182], 99.95th=[ 192],
| 99.99th=[ 207]
bw ( MiB/s): min= 2172, max= 4340, per=100.00%, avg=3507.33, stdev=293.91, samples=119
iops : min= 1086, max= 2170, avg=1753.49, stdev=147.00, samples=119
lat (msec) : 10=0.04%, 20=56.27%, 50=22.28%, 100=11.76%, 250=9.64%
cpu : usr=12.63%, sys=0.85%, ctx=113879, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,105079,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=3500MiB/s (3670MB/s), 3500MiB/s-3500MiB/s (3670MB/s-3670MB/s), io=205GiB (220GB), run=60048-60048msec

What I have tested:

  1. Different OSes/kernels
    Debian 13.x, Ubuntu 24.04.3 and 25.10
  2. Different zpool create options
    ashift, recordsize, atime, relatime, compression
  3. Formatting NVME LBA to 4k (it was 512 by default)
  4. Various ZFS versions
    including 2.3.0, 2.4.0-rc3, 2.3.4 from Debian/Ubuntu packages
  5. Different ARC sizes
  6. Different fio parameters
    iodepth, ioengine, bs

The only thing that improves seq. write performance on zfs-2.3.4 is disabling compression:


fio-3.39
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=4927MiB/s][w=2463 IOPS][eta 00m:00s]
test-seq-write: (groupid=0, jobs=1): err= 0: pid=51148: Sat Nov 15 19:37:14 2025
write: IOPS=2303, BW=4607MiB/s (4831MB/s)(270GiB/60023msec); 0 zone resets
slat (usec): min=3, max=375, avg=29.25, stdev=11.60
clat (msec): min=3, max=426, avg=27.75, stdev=28.94
lat (msec): min=3, max=426, avg=27.78, stdev=28.94
clat percentiles (msec):
| 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24],
| 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 24],
| 70.00th=[ 25], 80.00th=[ 25], 90.00th=[ 25], 95.00th=[ 26],
| 99.00th=[ 184], 99.50th=[ 271], 99.90th=[ 384], 99.95th=[ 401],
| 99.99th=[ 409]
bw ( MiB/s): min= 1228, max= 5544, per=99.99%, avg=4606.70, stdev=1153.87, samples=120
iops : min= 614, max= 2772, avg=2303.35, stdev=576.93, samples=120
lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=97.60%, 100=0.50%
lat (msec) : 250=1.21%, 500=0.64%
cpu : usr=6.87%, sys=0.47%, ctx=138486, majf=0, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,138264,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
WRITE: bw=4607MiB/s (4831MB/s), 4607MiB/s-4607MiB/s (4831MB/s-4831MB/s), io=270GiB (290GB), run=60023-60023msec

At the same time on zfs-2.2.8, disabling compression basically doesn’t change sequential write speeds.

I don’t know how to pinpoint the problem with LZ4 compression on zfs-2.3.4.
I’ve read somewhere that LZ4 have gotten an early abort feature starting from version 2.x/2.2.x?
Unfortunately, I don’t know the compression code or ZFS internals very well - just a gut feeling that maybe something broke in the way 2.3.x handles incompressible data, and now ZFS tries to compress it for too long instead of failing fast?
Also I couldn’t find any new parameters that change LZ4 behavior.

If something like that exists, please point me in the right direction.
I could also test source changes, maybe some thresholds/settings in the code that are not configurable have changed.
I would need guidance on what I can do to make compression work as well as it did on older ZFS 2.2.x.

Thanks and best regards!