Best ZFS settings for 24 SAMSUNG's MZWLJ1T9HBJR-00007

I have two PM9A3 U.2 and something is wrong when I format them with 4k, ZFS then complains with
“One or more devices are configured to use a non-native block size. Expect reduced performance.”
With any other NVME I have it’s no problem

Your result looks like your server is dead, but the test is also not useful, you should use fio

That’s only one PM9A3

# dd if=/dev/zero of=/tank02/test1.img bs=5G count=20 oflag=dsync
dd: warning: partial read (2147479552 bytes); suggest iflag=fullblock
0+20 Datensätze ein
0+20 Datensätze aus
42949591040 Bytes (43 GB, 40 GiB) kopiert, 4,7957 s, 9,0 GB/s

https://openzfs.github.io/openzfs-docs/Performance%20and%20Tuning/Workload%20Tuning.html