ZFS is really slow

Operating Systems & Open Source Linux
, ,
1

Hello,

I’m trying to figure out why copying from one zfs pool to another is really slow.

root@keksmoks:~# zpool iostat -v -l -q
                                                                                             capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim  rebuild   syncq_read    syncq_write   asyncq_read  asyncq_write   scrubq_read   trimq_write  rebuildq_write
pool                                                                                       alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait   wait   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ   pend  activ
-----------------------------------------------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
bulk01                                                                                     4.63T  60.8T      0      0     61   214K   10ms   62ms    8ms    8ms  833ns  457ns  684ns   54ms   17ms      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  raidz2-0                                                                                 4.63T  60.8T      0      0     61   214K   10ms   62ms    8ms    8ms  833ns  457ns  684ns   54ms   17ms      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-ST18000NM000J-XXXXXXXXXXXXXXX                                                          -      -      0      0     15  53.5K    9ms    7ms    8ms    1ms  845ns  557ns  614ns    5ms   12ms      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-ST18000NM000J-XXXXXXXXXXXXXXX                                                          -      -      0      0     15  53.5K   10ms  436ms    8ms   53ms  841ns  428ns  704ns  390ms   14ms      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-ST18000NM000J-XXXXXXXXXXXXXXX                                                          -      -      0      0     15  53.5K   10ms    7ms    8ms    1ms  819ns  402ns  713ns    6ms   17ms      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-ST18000NM000J-XXXXXXXXXXXXXXX                                                          -      -      0      0     15  53.5K    9ms    7ms    6ms    1ms  824ns  439ns  725ns    6ms   24ms      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
-----------------------------------------------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
rpool                                                                                      8.20G  1.85T      0     24  27.5K   269K  656us    4ms  599us  914us  541ns    4us  101us    3ms  811us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
  mirror-0                                                                                 8.20G  1.85T      0     24  27.5K   269K  656us    4ms  599us  914us  541ns    4us  101us    3ms  811us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    nvme-nvme.1987-3239364530373937313133373030303830383935-53616272656e74-00000001-part3      -      -      0     12  14.6K   134K  203us  619us  190us  131us  537ns    1us   18us  529us    1ms      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    nvme-ADATA_LEGEND_710_XXXXXXXXXXXX-part3                                                   -      -      0     11  12.8K   134K    1ms    8ms    1ms    1ms  545ns    8us  205us    6ms  156us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
-----------------------------------------------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
storagepool                                                                                5.38T  1.89T      6      0   101K  2.81K    3ms  296ms    3ms   63ms    1us  502ns  919us  258ms  327us      -      -      0      0      0      0      2      3     26      2      0      0      0      0      0      0
  raidz1-0                                                                                 5.38T  1.89T      6      0   101K  2.81K    3ms  296ms    3ms   63ms    1us  502ns  919us  258ms  327us      -      -      0      0      0      0      2      3     26      2      0      0      0      0      0      0
    ata-ADATA_SU800_2I4620029161                                                               -      -      0      0  12.9K    361  220us  243us  198us  100us  511ns  471ns   39us  158us  150us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-CT1000BX500SSD1_1944E226CB3A                                                           -      -      0      0  12.5K    359  472us  370us  406us  117us  813ns  642ns  131us  280us  283us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-Micron_1300_MTFDDAK1T0TDL_20192B1C5687                                                 -      -      0      0  13.1K    361   27ms     2s   23ms  514ms    4us  462ns    8ms     2s  269us      -      -      0      0      0      0      2      3     26      2      0      0      0      0      0      0
    ata-CT1000MX500SSD1_1920E2042A75                                                           -      -      0      0  12.0K    360  321us    7ms  275us    4ms  481ns  440ns   74us    3ms   61us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-SPCC_Solid_State_Disk_AA000000000000081149                                             -      -      0      0  13.0K    362  270us  222us  240us   82us  496ns  479ns   57us  156us  397us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-Micron_1300_MTFDDAK1T0TDL_1943248C0E09                                                 -      -      0      0  12.3K    360  289us  154us  258us   63us  533ns  512ns   55us  102us  751us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-MTFDDAK1T0TDL_200225E0E760                                                             -      -      0      0  13.0K    358  277us  170us  247us   64us  472ns  490ns   57us  115us  503us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
    ata-Micron_1300_MTFDDAK1T0TDL_20192AF8B71C                                                 -      -      0      0  12.2K    356  278us  166us  250us   64us  480ns  518ns   48us  113us  327us      -      -      0      0      0      0      0      0      0      0      0      0      0      0      0      0
-----------------------------------------------------------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----

In my htop I see not much disk activity and a lot of IOWait:

image
image3814×217 177 KB

I see that one of the disks is reported as disk_Wait -> write -> 514ms

So perhaps this SSD just wants to finally die or something, but I’d like to diagnose whether or not the disk is to blame or the controller or bad sata cable or perhaps something else.

How can I go about debugging this issue?

2

atop can also break down by disk. something like munin monitoring, or zabbix can also get more of a picture over time without you having to realtime monitor.

whichever drive it is, if it’s always lagging & if it’s not the only disk on the controller (many at least consumer level boards do use an asmedia on some of the board’s ports, so they’re not all chipset, even if not a consumer board, something similar could be going on) then you can just replace the cable (but for that smart might itself report udma errors, even if quite possibly zfs’s better crc would first).

if it stays suspect then I’d want to proactively replace it out to test independently.

i’ve seen some terrible outcomes with adata drives unfortunately (not super a fan of BX either. no judging).

1 Like
3

Screenshot_20260531_231533
Screenshot_20260531_2315336102×1562 1.82 MB

HMM…

Well, I think I have two sata available on the sas controller, gonna swtich them to that I guess

4

Ok, so the controller those two disks were on was one of the cheap sata controllers from aliexpress with lots of ports… I got it because it was cheap, I think we now know why lol.

After switching them to the SAS controller everything works.

2 Likes
5

They are usually made with a single 4 port controller with a x2 pcie 3.0 link and some sata port multipliers.

Sata port multipliers are not great for many reasons. If you were on truenas forums right now you would get shouted at for committing the cardinal sin of using sata port multipliers on zfs

I never buy sata controllers from sellers that don’t state what chipset model they are using. Asmedia makes controllers with 4 or 6 ports max and there is a strange sata controller from realtek with 9 ports. They are all on x2 pcie 3.0 so they are going to bottleneck ssds but will mostly be ok for mechanical drives.

1 Like