I’ve got a problem with sequential write performance on ZFS 2.3.x vs 2.2.x.
OS: Debian 13.1
NVME: 4 x Sandisk 2TB M.2 PCIe Gen5 NVMe Black SN8100
CPU: AMD EPYC 4564P
I’m testing by compiling ZFS from source, installing the generated *.deb files, and rebooting to make sure the kernel loads the new modules.
Every time I set these parameters:
echo "4294967295" > /sys/module/zfs/parameters/zfs_arc_min
echo "4294967296" > /sys/module/zfs/parameters/zfs_arc_max
Then I create the pool with:
zpool create -f \
-o ashift=12 \
-o autotrim=on \
-O acltype=posixacl -O xattr=sa -O dnodesize=auto \
-O compression=lz4 \
-O utf8only=off -O normalization=none \
-O relatime=on \
-O canmount=on -O mountpoint=/ztest -O snapdir=visible \
ztest mirror \
/dev/nvme0n1 \
/dev/nvme1n1 \
mirror \
/dev/nvme2n1 \
/dev/nvme3n1
Testing seq. read/write with fio:
fio --filename=/ztest/test.bin --direct=1 --rw=read --ioengine=io_uring --bs=2m --iodepth=64 --size=10G --numjobs=1 --runtime=60 --time_base=1 --group_reporting --name=test-seq-read --
fio --filename=/ztest/test.bin --direct=1 --rw=write --ioengine=io_uring --bs=2m --iodepth=64 --size=10G --numjobs=1 --runtime=60 --time_base=1 --group_reporting --name=test-seq-write --
Sequential read performance shows a big improvement:
zfs v2.3.4:
fio-3.39
Starting 1 process
test-seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=19.5GiB/s][r=9962 IOPS][eta 00m:00s]
test-seq-read: (groupid=0, jobs=1): err= 0: pid=229665: Thu Nov 13 13:37:59 2025
read: IOPS=9966, BW=19.5GiB/s (20.9GB/s)(1168GiB/60006msec)
slat (nsec): min=331, max=3970.8k, avg=3094.65, stdev=17943.07
clat (usec): min=92, max=24643, avg=6416.33, stdev=644.56
lat (usec): min=281, max=24657, avg=6419.43, stdev=644.38
clat percentiles (usec):
| 1.00th=[ 4424], 5.00th=[ 5604], 10.00th=[ 5932], 20.00th=[ 6194],
| 30.00th=[ 6325], 40.00th=[ 6390], 50.00th=[ 6456], 60.00th=[ 6521],
| 70.00th=[ 6587], 80.00th=[ 6652], 90.00th=[ 6783], 95.00th=[ 6915],
| 99.00th=[ 8356], 99.50th=[ 9503], 99.90th=, 99.95th=,
| 99.99th=
bw ( MiB/s): min=19144, max=20160, per=100.00%, avg=19943.02, stdev=84.88, samples=119
iops : min= 9572, max=10080, avg=9971.50, stdev=42.43, samples=119
lat (usec) : 100=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.06%, 4=0.63%, 10=98.93%, 20=0.37%, 50=0.01%
cpu : usr=1.57%, sys=6.86%, ctx=552160, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=598043,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=19.5GiB/s (20.9GB/s), 19.5GiB/s-19.5GiB/s (20.9GB/s-20.9GB/s), io=1168GiB (1254GB), run=60006-60006msec
zfs v2.2.8:
fio-3.39
Starting 1 process
test-seq-read: Laying out IO file (1 file / 10240MiB)
Jobs: 1 (f=1): [R(1)][100.0%][r=5016MiB/s][r=2507 IOPS][eta 00m:00s]
test-seq-read: (groupid=0, jobs=1): err= 0: pid=102593: Sat Nov 15 00:37:30 2025
read: IOPS=2512, BW=5026MiB/s (5270MB/s)(295GiB/60013msec)
slat (nsec): min=330, max=11393k, avg=22868.39, stdev=197445.05
clat (usec): min=220, max=167763, avg=25434.27, stdev=13153.14
lat (usec): min=519, max=167768, avg=25457.14, stdev=13150.89
clat percentiles (msec):
| 1.00th=[ 8], 5.00th=[ 11], 10.00th=[ 12], 20.00th=[ 15],
| 30.00th=[ 18], 40.00th=[ 20], 50.00th=[ 23], 60.00th=[ 26],
| 70.00th=[ 29], 80.00th=[ 35], 90.00th=[ 43], 95.00th=[ 52],
| 99.00th=[ 70], 99.50th=[ 78], 99.90th=[ 94], 99.95th=[ 102],
| 99.99th=[ 121]
bw ( MiB/s): min= 4892, max= 5137, per=100.00%, avg=5029.29, stdev=50.95, samples=119
iops : min= 2446, max= 2568, avg=2514.25, stdev=25.32, samples=119
lat (usec) : 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01%
lat (msec) : 2=0.01%, 4=0.01%, 10=4.47%, 20=36.30%, 50=53.59%
lat (msec) : 100=5.57%, 250=0.06%
cpu : usr=2.96%, sys=4.71%, ctx=61753, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=150799,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
READ: bw=5026MiB/s (5270MB/s), 5026MiB/s-5026MiB/s (5270MB/s-5270MB/s), io=295GiB (316GB), run=60013-60013msec
But something got seriously broken with sequential write:
zfs v2.3.4:
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=936MiB/s][w=468 IOPS][eta 00m:00s]
test-seq-write: (groupid=0, jobs=1): err= 0: pid=229787: Thu Nov 13 13:39:44 2025
write: IOPS=469, BW=939MiB/s (984MB/s)(55.1GiB/60135msec); 0 zone resets
slat (nsec): min=3386, max=98575, avg=25828.94, stdev=8272.96
clat (msec): min=3, max=266, avg=136.32, stdev= 8.07
lat (msec): min=3, max=266, avg=136.35, stdev= 8.07
clat percentiles (msec):
| 1.00th=[ 112], 5.00th=[ 136], 10.00th=[ 136], 20.00th=[ 136],
| 30.00th=[ 136], 40.00th=[ 136], 50.00th=[ 138], 60.00th=[ 138],
| 70.00th=[ 138], 80.00th=[ 138], 90.00th=[ 138], 95.00th=[ 140],
| 99.00th=[ 144], 99.50th=[ 178], 99.90th=[ 209], 99.95th=[ 236],
| 99.99th=[ 264]
bw ( KiB/s): min=819200, max=1257472, per=100.00%, avg=961228.80, stdev=34982.79, samples=120
iops : min= 400, max= 614, avg=469.35, stdev=17.08, samples=120
lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=0.08%, 100=0.39%
lat (msec) : 250=99.46%, 500=0.03%
cpu : usr=1.27%, sys=0.08%, ctx=28254, majf=0, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.8%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,28224,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=939MiB/s (984MB/s), 939MiB/s-939MiB/s (984MB/s-984MB/s), io=55.1GiB (59.2GB), run=60135-60135msec
zfs v2.2.8:
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=3341MiB/s][w=1670 IOPS][eta 00m:00s]
test-seq-write: (groupid=0, jobs=1): err= 0: pid=102725: Sat Nov 15 00:39:46 2025
write: IOPS=1749, BW=3500MiB/s (3670MB/s)(205GiB/60048msec); 0 zone resets
slat (usec): min=4, max=56978, avg=109.01, stdev=564.70
clat (msec): min=4, max=220, avg=36.45, stdev=35.00
lat (msec): min=4, max=220, avg=36.56, stdev=35.07
clat percentiles (msec):
| 1.00th=[ 16], 5.00th=[ 16], 10.00th=[ 16], 20.00th=[ 16],
| 30.00th=[ 17], 40.00th=[ 18], 50.00th=[ 20], 60.00th=[ 21],
| 70.00th=[ 29], 80.00th=[ 56], 90.00th=[ 99], 95.00th=[ 126],
| 99.00th=[ 138], 99.50th=[ 142], 99.90th=[ 182], 99.95th=[ 192],
| 99.99th=[ 207]
bw ( MiB/s): min= 2172, max= 4340, per=100.00%, avg=3507.33, stdev=293.91, samples=119
iops : min= 1086, max= 2170, avg=1753.49, stdev=147.00, samples=119
lat (msec) : 10=0.04%, 20=56.27%, 50=22.28%, 100=11.76%, 250=9.64%
cpu : usr=12.63%, sys=0.85%, ctx=113879, majf=0, minf=7
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,105079,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=3500MiB/s (3670MB/s), 3500MiB/s-3500MiB/s (3670MB/s-3670MB/s), io=205GiB (220GB), run=60048-60048msec
What I have tested:
- Different OSes/kernels
Debian 13.x, Ubuntu 24.04.3 and 25.10 - Different zpool create options
ashift, recordsize, atime, relatime, compression - Formatting NVME LBA to 4k (it was 512 by default)
- Various ZFS versions
including 2.3.0, 2.4.0-rc3, 2.3.4 from Debian/Ubuntu packages - Different ARC sizes
- Different fio parameters
iodepth, ioengine, bs
The only thing that improves seq. write performance on zfs-2.3.4 is disabling compression:
fio-3.39
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=4927MiB/s][w=2463 IOPS][eta 00m:00s]
test-seq-write: (groupid=0, jobs=1): err= 0: pid=51148: Sat Nov 15 19:37:14 2025
write: IOPS=2303, BW=4607MiB/s (4831MB/s)(270GiB/60023msec); 0 zone resets
slat (usec): min=3, max=375, avg=29.25, stdev=11.60
clat (msec): min=3, max=426, avg=27.75, stdev=28.94
lat (msec): min=3, max=426, avg=27.78, stdev=28.94
clat percentiles (msec):
| 1.00th=[ 23], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24],
| 30.00th=[ 24], 40.00th=[ 24], 50.00th=[ 24], 60.00th=[ 24],
| 70.00th=[ 25], 80.00th=[ 25], 90.00th=[ 25], 95.00th=[ 26],
| 99.00th=[ 184], 99.50th=[ 271], 99.90th=[ 384], 99.95th=[ 401],
| 99.99th=[ 409]
bw ( MiB/s): min= 1228, max= 5544, per=99.99%, avg=4606.70, stdev=1153.87, samples=120
iops : min= 614, max= 2772, avg=2303.35, stdev=576.93, samples=120
lat (msec) : 4=0.01%, 10=0.01%, 20=0.02%, 50=97.60%, 100=0.50%
lat (msec) : 250=1.21%, 500=0.64%
cpu : usr=6.87%, sys=0.47%, ctx=138486, majf=0, minf=6
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,138264,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=4607MiB/s (4831MB/s), 4607MiB/s-4607MiB/s (4831MB/s-4831MB/s), io=270GiB (290GB), run=60023-60023msec
At the same time on zfs-2.2.8, disabling compression basically doesn’t change sequential write speeds.
I don’t know how to pinpoint the problem with LZ4 compression on zfs-2.3.4.
I’ve read somewhere that LZ4 have gotten an early abort feature starting from version 2.x/2.2.x?
Unfortunately, I don’t know the compression code or ZFS internals very well - just a gut feeling that maybe something broke in the way 2.3.x handles incompressible data, and now ZFS tries to compress it for too long instead of failing fast?
Also I couldn’t find any new parameters that change LZ4 behavior.
If something like that exists, please point me in the right direction.
I could also test source changes, maybe some thresholds/settings in the code that are not configurable have changed.
I would need guidance on what I can do to make compression work as well as it did on older ZFS 2.2.x.
Thanks and best regards!