Link speed slower over fiber SFP28 vs cooper SFP+?

Hardware Hub Networking
, ,
41

I think I figured out 1 piece of info that could be the culprit for the slow network transfer rates.

Local file copy in Windows

On my desktop, when copying from one NVMe drive to the other, the transfer speed is horribly slow. Both of these are 4TB Corsair MP600 PRO XT PCIe 4.0 NVMe drives running in two PCIe 4.0 M.2 slots.

HWiNFO shows both drives running at NVMe (PCIe 4x 16.0 GT/s).

I ran a TRIM operation on them just now to be sure and got the same results:
image

The drives aren’t full either.
image

D: ↔️ C:

This one started out fast, but then quickly dropped:

image
image449×287 6.35 KB

C: ↔️ D:

image
image449×287 6.34 KB

In the past, I tested file copies between these drives at over 2GB/s, so I didn’t think to check again. I also ran CrystalDiskMark in the past and got really fast speeds all-around. I’m honestly surprised.

CrystalDiskMark

C:

image
image799×181 71.5 KB

D:

No clue why this drive is so much slower considering it’s not running Windows.

image
image800×179 72.1 KB

Are file transfers in Windows 11’s Explorer completely broken?

The slow file transfer writes explains why reading NAS data is so slow. It’s copying to these drives at the same slow speed they copy to each other. That explains why CrystalDiskMark over SMB to the zpool was showing 1.2Gb reads versus my 200-600MB/s file copies off the NAS.

But this doesn’t explain why writes were 500MB/s slower to the NAS using 25Gb vs 10Gb even with CrystalDiskMark. So that issue still remains.

Questions I had for this thread:

  1. :white_check_mark: Read speed slower than write speed over either connection.
  2. :question: Write speed over 10Gb faster than 25Gb.
42

Oh my gosh.

What is this journalctl -r | grep -i error command? There’s so much stuff that it took 30 sec for my CTRL-C to kill it. I should’ve done tail -n 100 or less.

I wish I could have a 10 year old system full of 10 year old HDDs. Sounds like that’s the sweet spot for performance :stuck_out_tongue:. I’ll take 60TB over my 30TB any day if I can get more than double the performance out of it.

Looks like TrueNAS apps are having some issues. Those are installed into a different zpool, so they shouldn’t be affecting these, but maybe this is affecting overall compute time. What else can I grep to get SSD-related data?

Feb 13 01:13:26 storeman k3s[27082]: E0213 01:13:26.344501   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.275930   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-resizer/0.log\": failed to reopen container log \"cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-resizer/0.log" containerID="cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.275756   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.271721   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/snapshot-controller/0.log\": failed to reopen container log \"ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/snapshot-controller/0.log" containerID="ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.271544   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.269419   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-provisioner/0.log\": failed to reopen container log \"52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-provisioner/0.log" containerID="52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.269210   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.261274   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-xkhvq_7185ac57-9b29-4ea6-842a-872adb65d6e3/openebs-zfs-plugin/0.log\": failed to reopen container log \"ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-xkhvq_7185ac57-9b29-4ea6-842a-872adb65d6e3/openebs-zfs-plugin/0.log" containerID="ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779"
Feb 13 01:13:16 storeman k3s[27082]: E0213 01:13:16.261127   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.193866   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-resizer/0.log\": failed to reopen container log \"cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-resizer/0.log" containerID="cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.193683   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.189654   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/snapshot-controller/0.log\": failed to reopen container log \"ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/snapshot-controller/0.log" containerID="ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.189490   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.187145   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-provisioner/0.log\": failed to reopen container log \"52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-provisioner/0.log" containerID="52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.186929   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.179419   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-xkhvq_7185ac57-9b29-4ea6-842a-872adb65d6e3/openebs-zfs-plugin/0.log\": failed to reopen container log \"ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-xkhvq_7185ac57-9b29-4ea6-842a-872adb65d6e3/openebs-zfs-plugin/0.log" containerID="ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779"
Feb 13 01:13:06 storeman k3s[27082]: E0213 01:13:06.179196   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779"
Feb 13 01:12:56 storeman k3s[27082]: E0213 01:12:56.111324   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-resizer/0.log\": failed to reopen container log \"cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-resizer/0.log" containerID="cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805"
Feb 13 01:12:56 storeman k3s[27082]: E0213 01:12:56.111165   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="cc045183cbe3dc2f527e2164ce61254c68cc36d2d5e8660bd8f05878fb6fd805"
Feb 13 01:12:56 storeman k3s[27082]: E0213 01:12:56.107025   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/snapshot-controller/0.log\": failed to reopen container log \"ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/snapshot-controller/0.log" containerID="ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29"
Feb 13 01:12:56 storeman k3s[27082]: E0213 01:12:56.106775   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="ece5d0591c915ddfa4cc788afe0c56a9a3286f06f6a8f1563d8ea6239e0d7c29"
Feb 13 01:12:56 storeman k3s[27082]: E0213 01:12:56.104509   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-provisioner/0.log\": failed to reopen container log \"52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_e9098562-c300-4edb-9f03-f8de1460a8c0/csi-provisioner/0.log" containerID="52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474"
Feb 13 01:12:56 storeman k3s[27082]: E0213 01:12:56.104354   27082 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="52732e165647ca45f487b68fc0f21c83e979d0fdfb9354c3f2e1ba0c779af474"
Feb 13 01:12:56 storeman k3s[27082]: E0213 01:12:56.096332   27082 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-xkhvq_7185ac57-9b29-4ea6-842a-872adb65d6e3/openebs-zfs-plugin/0.log\": failed to reopen container log \"ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc8c789b779\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-xkhvq_7185ac57-9b29-4ea6-842a-872adb65d6e3/openebs-zfs-plugin/0.log" containerID="ce458ccf34b871dacf7be9a5cdf11eadca4bd87a7019478e5e827bc
43

with “error” you should see all errors, also those of an SSD.
Delete your appstore, if not used and grep again

Use what you want, for example Bios

root@pve-backup:~# journalctl -b1 | grep -i Bios
Jun 23 01:01:25 pve-backup kernel: BIOS-provided physical RAM map:
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x0000000000000000-0x000000000009c7ff] usable
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x000000000009c800-0x000000000009ffff] reserved
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x0000000000100000-0x00000000bddabfff] usable
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x00000000bddac000-0x00000000bddddfff] ACPI data
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x00000000bddde000-0x00000000cfffffff] reserved
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x00000000fec00000-0x00000000fee0ffff] reserved
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x00000000ff800000-0x00000000ffffffff] reserved
Jun 23 01:01:25 pve-backup kernel: BIOS-e820: [mem 0x0000000100000000-0x000000203fffefff] usable
Jun 23 01:01:25 pve-backup kernel: SMBIOS 2.8 present.
Jun 23 01:01:25 pve-backup kernel: DMI: HP ProLiant DL380p Gen8, BIOS P70 05/21/2018
Jun 23 01:01:25 pve-backup kernel: ACPI BIOS Warning (bug): Invalid length for FADT/Pm1aControlBlock: 32, using default 16 (20210730/tbfadt-669)
Jun 23 01:01:25 pve-backup kernel: ACPI BIOS Warning (bug): Invalid length for FADT/Pm2ControlBlock: 32, using default 8 (20210730/tbfadt-669)
Jun 23 01:01:25 pve-backup kernel: DMAR: [Firmware Bug]: No firmware reserved region can cover this RMRR [0x00000000000e8000-0x00000000000e8fff], contact BIOS vendor for fixes
Jun 23 01:01:25 pve-backup kernel: DMAR: [Firmware Bug]: Your BIOS is broken; bad RMRR [0x00000000000e8000-0x00000000000e8fff]
BIOS vendor: HP; Ver: P70; Product Version:
Jun 23 01:01:25 pve-backup kernel: DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
Jun 23 01:01:25 pve-backup kernel: DMAR-IR: Use ‘intremap=no_x2apic_optout’ to override the BIOS setting.
Jun 23 01:01:25 pve-backup kernel: Performance Events: PEBS fmt1+, SandyBridge events, 16-deep LBR, full-width counters, Broken BIOS detected, complain to your hardware vendor.
Jun 23 01:01:25 pve-backup kernel: [Firmware Bug]: the BIOS has corrupted hw-PMU resources (MSR 38d is 330)
Jun 23 01:01:25 pve-backup kernel: mpt3sas_cm0: LSISAS3008: FWVersion(16.00.10.00), ChipRevision(0x02), BiosVersion(08.37.02.00)
Jun 23 01:01:25 pve-backup kernel: ipmi_si dmi-ipmi-si.0: ipmi_platform: probing via SMBIOS
Jun 23 01:01:25 pve-backup kernel: ipmi_platform: ipmi_si: SMBIOS: io 0xca2 regsize 1 spacing 1 irq 0
Jun 23 01:01:25 pve-backup kernel: ipmi_si: Adding SMBIOS-specified kcs state machine
Jun 23 01:01:25 pve-backup kernel: ipmi_si dmi-ipmi-si.0: Removing SMBIOS-specified kcs state machine in favor of ACPI

44

I found a way to exclude unnecessary stuff from the grep.

What looks important

Both sdbi and sdbj are part of the boot-pool for TrueNAS. They’re also directly connected via SATA through a SlimSAS cable. So they’re not on the LSI cards.

I could just as well swap the drives if they’re being problematic. I bought these 60GB Corsair Force GT as refurbs in 2014, so it wouldn’t surprise me if they’re sketch.

Either way, I don’t think these issues harm my main array.

Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 10 02:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 102 to 104
Feb 10 02:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 102 to 104
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 104
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 104
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 102
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 102

Full log

I had DVD and Blu-Ray drives connected to rip some discs for Plex. Those aren’t connected anymore which explains the /dev/sr* errors.

# journalctl -r | grep -i error | grep -wv kube | grep -wv k3s
Feb 15 04:29:12 storeman dockerd[26348]: time="2023-02-15T04:29:12.183805455-06:00" level=error msg="Unable to determine real path of /dev/sr2 for validation"
Feb 15 04:29:12 storeman dockerd[26348]: time="2023-02-15T04:29:12.183794995-06:00" level=error msg="Unable to determine real path of /dev/sr1 for validation"
Feb 15 04:29:12 storeman dockerd[26348]: time="2023-02-15T04:29:12.183765224-06:00" level=error msg="Unable to determine real path of /dev/sr0 for validation"
Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 14 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 106
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdaf [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbc [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdai [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbe [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdq [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdw [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sday [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdf [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdt [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdm [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdaj [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdae [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdab [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdav [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdat [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdaq [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbd [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdba [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdaw [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdu [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdak [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbg [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdax [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdv [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdp [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdk [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdx [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbk [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbl [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdr [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdi [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdb [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdap [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdaa [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdaz [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdao [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdc [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdg [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdn [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdj [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sda [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdd [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sde [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdo [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdh [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdam [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdz [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdan [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdas [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdar [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdl [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdy [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdad [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbb [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbh [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdag [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sds [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdau [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdah [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdac [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdal [SAT], previous self-test completed without error
Feb 14 00:54:07 storeman smartd[21882]: Device: /dev/sdbf [SAT], previous self-test completed without error
Feb 13 22:16:15 storeman kernel: sr 24:0:0:0: tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
Feb 13 22:16:15 storeman kernel: sr 23:0:0:0: tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
Feb 13 22:16:15 storeman kernel: sr 23:0:0:0: tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
Feb 13 22:16:15 storeman kernel: sr 26:0:0:0: tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
Feb 13 22:16:14 storeman kernel: sr 26:0:0:0: tag#0 FAILED Result: hostbyte=DID_ERROR driverbyte=DRIVER_OK cmd_age=0s
Feb 13 21:04:16 storeman nginx[22107]: 2023/02/13 21:04:16 [error] 22107#22107: *528 recv() failed (104: Connection reset by peer) while proxying upgraded connection, client: 10.1.0.228, server: localhost, request: "GET /websocket/shell/ HTTP/1.1", upstream: "http://127.0.0.1:6000/_shell/", host: "storeman.octen"
Feb 13 20:45:35 storeman dockerd[26406]: time="2023-02-13T20:45:35.244183105-06:00" level=warning msg="cleanup warnings time=\"2023-02-13T20:45:35-06:00\" level=info msg=\"starting signal loop\" namespace=moby pid=4109014 runtime=io.containerd.runc.v2\ntime=\"2023-02-13T20:45:35-06:00\" level=warning msg=\"failed to remove runc container\" error=\"runc did not terminate successfully: exit status 255: \" runtime=io.containerd.runc.v2\n"
Feb 13 20:39:52 storeman kernel: blk_update_request: I/O error, dev sr1, sector 10567200 op 0x0:(READ) flags 0x0 phys_seg 28 prio class 0
Feb 13 20:39:52 storeman kernel: blk_update_request: I/O error, dev sr1, sector 10566688 op 0x0:(READ) flags 0x0 phys_seg 24 prio class 0
Feb 13 20:29:09 storeman kernel: Buffer I/O error on dev sr1, logical block 1320834, async page read
Feb 13 20:29:09 storeman kernel: blk_update_request: I/O error, dev sr1, sector 10566672 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 13 20:29:09 storeman kernel: blk_update_request: I/O error, dev sr1, sector 10566656 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
Feb 11 03:11:23 storeman wsdd.py[2132892]: OSError: [Errno 98] Address already in use
Feb 11 03:11:23 storeman wsdd.py[2132892]: ERROR: error in main loop
Feb 11 02:56:42 storeman wsdd.py[2076509]: OSError: [Errno 98] Address already in use
Feb 11 02:56:42 storeman wsdd.py[2076509]: ERROR: error in main loop
Feb 10 02:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 102 to 104
Feb 10 02:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 102 to 104
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 104
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 104
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 102
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 100 to 120
Feb 07 03:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 120 to 102
Feb 07 00:54:08 storeman smartd[21882]: Device: /dev/sdaf [SAT], previous self-test completed without error
Feb 07 00:54:08 storeman smartd[21882]: Device: /dev/sdbc [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdai [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbe [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdq [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdw [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sday [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdf [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdt [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdm [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdaj [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdae [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdab [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdav [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdat [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdaq [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbd [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdba [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdaw [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdu [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdak [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbg [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdax [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdv [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdp [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdk [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdx [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbk [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbl [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdr [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdi [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdb [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdap [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdaa [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdaz [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdao [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdc [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdg [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdn [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdj [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sda [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdd [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sde [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdo [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdh [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdam [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdz [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdan [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdas [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdar [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdl [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdy [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdad [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbb [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbh [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdag [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sds [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdau [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdah [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbi [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdac [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbj [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdal [SAT], previous self-test completed without error
Feb 07 00:54:07 storeman smartd[21882]: Device: /dev/sdbf [SAT], previous self-test completed without error
Feb 05 03:56:28 storeman systemd-xdg-autostart-generator[2282874]: Not generating service for XDG autostart app-at\x2dspi\x2ddbus\x2dbus-autostart.service, error parsing Exec= line: No such file or directory
Feb 05 00:18:45 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819116 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 05 00:18:45 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:18:45 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 05 00:18:10 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819116 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 05 00:18:10 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:18:10 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 05 00:17:37 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819116 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 05 00:17:37 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:17:37 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 05 00:17:06 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819116 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 05 00:17:06 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:17:06 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 05 00:16:33 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819116 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 05 00:16:33 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:16:33 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 05 00:15:12 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819836 op 0x0:(READ) flags 0x0 phys_seg 5 prio class 0
Feb 05 00:15:12 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:15:12 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 05 00:14:36 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819596 op 0x0:(READ) flags 0x4000 phys_seg 11 prio class 0
Feb 05 00:14:36 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:14:36 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 05 00:14:06 storeman kernel: blk_update_request: I/O error, dev sr0, sector 7819356 op 0x0:(READ) flags 0x4000 phys_seg 17 prio class 0
Feb 05 00:14:06 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Add. Sense: Unrecovered read error
Feb 05 00:14:06 storeman kernel: sr 23:0:0:0: [sr0] tag#0 Sense Key : Medium Error [deferred] 
Feb 04 23:59:32 storeman kernel: Buffer I/O error on dev sr2, logical block 3226645, async page read
Feb 04 23:59:32 storeman kernel: blk_update_request: I/O error, dev sr2, sector 12906580 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:59:32 storeman kernel: Buffer I/O error on dev sr2, logical block 3226644, async page read
Feb 04 23:59:32 storeman kernel: blk_update_request: I/O error, dev sr2, sector 12906576 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:59:31 storeman kernel: blk_update_request: I/O error, dev sr2, sector 12906496 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
Feb 04 23:37:05 storeman kernel: Buffer I/O error on dev sr1, logical block 965120, async page read
Feb 04 23:37:05 storeman kernel: blk_update_request: I/O error, dev sr1, sector 7720960 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:37:05 storeman kernel: blk_update_request: I/O error, dev sr1, sector 7720960 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
Feb 04 23:23:10 storeman kernel: Buffer I/O error on dev sr2, logical block 1478210, async page read
Feb 04 23:23:10 storeman kernel: blk_update_request: I/O error, dev sr2, sector 11825680 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:23:10 storeman kernel: blk_update_request: I/O error, dev sr2, sector 11825664 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
Feb 04 23:23:08 storeman kernel: Buffer I/O error on dev sr1, logical block 1055362, async page read
Feb 04 23:23:08 storeman kernel: blk_update_request: I/O error, dev sr1, sector 8442896 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:23:07 storeman kernel: blk_update_request: I/O error, dev sr1, sector 8442880 op 0x0:(READ) flags 0x80700 phys_seg 64 prio class 0
Feb 04 23:23:06 storeman kernel: Buffer I/O error on dev sr1, logical block 1024, async page read
Feb 04 23:23:06 storeman kernel: blk_update_request: I/O error, dev sr1, sector 8192 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:23:06 storeman kernel: blk_update_request: I/O error, dev sr1, sector 8192 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Feb 04 23:07:37 storeman kernel: Buffer I/O error on dev sr0, logical block 1107520, async page read
Feb 04 23:07:37 storeman kernel: blk_update_request: I/O error, dev sr0, sector 8860160 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:07:37 storeman kernel: blk_update_request: I/O error, dev sr0, sector 8860160 op 0x0:(READ) flags 0x80700 phys_seg 30 prio class 0
Feb 04 23:07:18 storeman kernel: Buffer I/O error on dev sr0, logical block 1024, async page read
Feb 04 23:07:18 storeman kernel: blk_update_request: I/O error, dev sr0, sector 8192 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:07:18 storeman kernel: blk_update_request: I/O error, dev sr0, sector 8192 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Feb 04 23:01:57 storeman kernel: Buffer I/O error on dev sr2, logical block 843648, async page read
Feb 04 23:01:57 storeman kernel: blk_update_request: I/O error, dev sr2, sector 6749184 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:01:57 storeman kernel: blk_update_request: I/O error, dev sr2, sector 6749184 op 0x0:(READ) flags 0x80700 phys_seg 60 prio class 0
Feb 04 23:01:56 storeman kernel: Buffer I/O error on dev sr2, logical block 256, async page read
Feb 04 23:01:56 storeman kernel: blk_update_request: I/O error, dev sr2, sector 2048 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 23:01:56 storeman kernel: blk_update_request: I/O error, dev sr2, sector 2048 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Feb 04 22:36:51 storeman kernel: Buffer I/O error on dev sr2, logical block 1069184, async page read
Feb 04 22:36:51 storeman kernel: blk_update_request: I/O error, dev sr2, sector 8553472 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:36:51 storeman kernel: blk_update_request: I/O error, dev sr2, sector 8553472 op 0x0:(READ) flags 0x80700 phys_seg 62 prio class 0
Feb 04 22:29:49 storeman ntpd[22012]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Feb 04 22:25:33 storeman wsdd.py[26231]: OSError: [Errno 98] Address already in use
Feb 04 22:25:33 storeman wsdd.py[26231]: ERROR: error in main loop
Feb 04 22:25:00 storeman dockerd[26348]: time="2023-02-04T22:25:00.655005028-06:00" level=error msg="Unable to determine real path of /dev/sr3 for validation"
Feb 04 22:24:23 storeman dockerd[26348]: time="2023-02-04T22:24:23.564744246-06:00" level=error msg="Unable to determine real path of /var/lib/kubelet/pods/93844450-1911-48a8-ae94-bffe1185202d/volumes/kubernetes.io~empty-dir/tailscale-manifests-temp for validation"
Feb 04 22:24:23 storeman dockerd[26348]: time="2023-02-04T22:24:23.550296992-06:00" level=error msg="Unable to determine real path of /var/lib/kubelet/pods/551b1a84-b3f3-4275-89bd-379b0f5bd854/containers/zerotier-manifests/e50e6679 for validation"
Feb 04 22:24:23 storeman dockerd[26348]: time="2023-02-04T22:24:23.550279182-06:00" level=error msg="Unable to determine real path of /var/lib/kubelet/pods/551b1a84-b3f3-4275-89bd-379b0f5bd854/etc-hosts for validation"
Feb 04 22:24:23 storeman dockerd[26348]: time="2023-02-04T22:24:23.550199092-06:00" level=error msg="Unable to determine real path of /var/lib/kubelet/pods/551b1a84-b3f3-4275-89bd-379b0f5bd854/volumes/kubernetes.io~empty-dir/zerotier-manifests-temp for validation"
Feb 04 22:24:23 storeman dockerd[26406]: time="2023-02-04T22:24:23.352062294-06:00" level=error msg="failed to initialize a tracing processor \"otlp\"" error="no OpenTelemetry endpoint: skip plugin"
Feb 04 22:24:23 storeman dockerd[26406]: time="2023-02-04T22:24:23.351961884-06:00" level=info msg="skip loading plugin \"io.containerd.tracing.processor.v1.otlp\"..." error="no OpenTelemetry endpoint: skip plugin" type=io.containerd.tracing.processor.v1
Feb 04 22:24:23 storeman dockerd[26406]: time="2023-02-04T22:24:23.349013223-06:00" level=warning msg="could not use snapshotter devmapper in metadata plugin" error="devmapper not configured"
Feb 04 22:24:23 storeman dockerd[26406]: time="2023-02-04T22:24:23.328291476-06:00" level=warning msg="failed to load plugin io.containerd.snapshotter.v1.devmapper" error="devmapper not configured"
Feb 04 22:24:23 storeman dockerd[26406]: time="2023-02-04T22:24:23.328254236-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.btrfs\"..." error="path /mnt/TrueNAS-Apps/ix-applications/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs (zfs) must be a btrfs filesystem to be used with the btrfs snapshotter: skip plugin" type=io.containerd.snapshotter.v1
Feb 04 22:24:23 storeman dockerd[26406]: time="2023-02-04T22:24:23.327351426-06:00" level=info msg="skip loading plugin \"io.containerd.snapshotter.v1.aufs\"..." error="aufs is not supported (modprobe aufs failed: exit status 1 \"modprobe: FATAL: Module aufs not found in directory /lib/modules/5.15.79+truenas\\n\"): skip plugin" type=io.containerd.snapshotter.v1
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 120 to 100
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 105 to 120
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 120 to 100
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbi [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 105 to 120
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 120 to 100
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 120
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Usage Attribute: 195 ECC_Uncorr_Error_Count changed from 120 to 100
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbj [SAT], SMART Prefailure Attribute: 1 Raw_Read_Error_Rate changed from 104 to 120
Feb 04 22:24:06 storeman syslog-ng[21940]: [2023-02-04T22:24:06.031152] Current position not matches to the saved cursor position, seek to head; cursor='s=e593f3055c7345a2b8440589dcda714c;i=1c86;b=40a922ca176b4d1cbdb32d5f00a0c69d;m=94d1333;t=5f3ec3523c798;x=29b452f3f03991d3', error='Success (0)'
Feb 04 22:24:06 storeman smartd[21882]: Device: /dev/sdbi [SAT], no SMART Error Log, ignoring -l error (override with -T permissive)
Feb 04 22:24:05 storeman smartd[21882]: Device: /dev/sdbi [SAT], no SMART Error Log, ignoring -l error (override with -T permissive)
Feb 04 22:24:05 storeman smartd[21882]: Device: /dev/sdbj [SAT], no SMART Error Log, ignoring -l error (override with -T permissive)
Feb 04 22:24:05 storeman ntpd[22012]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Feb 04 22:24:05 storeman ntpd[22012]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized
Feb 04 22:24:05 storeman smartd[21882]: Device: /dev/sdbj [SAT], no SMART Error Log, ignoring -l error (override with -T permissive)
Feb 04 22:23:19 storeman kernel: Error: Driver 'pcspkr' is already registered, aborting...
Feb 04 22:23:19 storeman kernel: Buffer I/O error on dev sr1, logical block 512, async page read
Feb 04 22:23:19 storeman kernel: blk_update_request: I/O error, dev sr1, sector 4096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:19 storeman kernel: Buffer I/O error on dev sr1, logical block 2044890, async page read
Feb 04 22:23:19 storeman kernel: blk_update_request: I/O error, dev sr1, sector 16359120 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:19 storeman kernel: blk_update_request: I/O error, dev sr1, sector 16358912 op 0x0:(READ) flags 0x80700 phys_seg 5 prio class 0
Feb 04 22:23:18 storeman kernel: Buffer I/O error on dev sr1, logical block 512, async page read
Feb 04 22:23:18 storeman kernel: buffer_io_error: 969 callbacks suppressed
Feb 04 22:23:18 storeman kernel: blk_update_request: I/O error, dev sr1, sector 4096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:18 storeman kernel: blk_update_request: I/O error, dev sr1, sector 4096 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Feb 04 22:23:18 storeman kernel: print_req_error: 458 callbacks suppressed
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr2, sector 0 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: blk_update_request: critical target error, dev sr2, sector 0 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr2, logical block 2097142, async page read
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr2, logical block 2097141, async page read
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr2, logical block 2097140, async page read
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr2, logical block 2097139, async page read
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr2, logical block 2097138, async page read
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr2, logical block 2097137, async page read
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr2, sector 2097137 op 0x0:(READ) flags 0x0 phys_seg 7 prio class 0
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr2, logical block 2097136, async page read
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr2, sector 2097136 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: blk_update_request: critical target error, dev sr2, sector 2097136 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr1, logical block 512, async page read
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr1, sector 4096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr1, logical block 2044890, async page read
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr1, sector 16359120 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr1, sector 16358912 op 0x0:(READ) flags 0x80700 phys_seg 59 prio class 0
Feb 04 22:23:17 storeman kernel: Buffer I/O error on dev sr1, logical block 512, async page read
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr1, sector 4096 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: blk_update_request: I/O error, dev sr1, sector 4096 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 0
Feb 04 22:23:17 storeman kernel: usb 5-2.1.2.4: device not accepting address 9, error -71
Feb 04 22:23:17 storeman kernel: usb 5-2.1.2.4: device not accepting address 8, error -71
Feb 04 22:23:17 storeman kernel: usb 5-2.1.2.4: device descriptor read/64, error -71
Feb 04 22:23:17 storeman kernel: usb 5-2.1.2.4: device descriptor read/64, error -71
Feb 04 22:23:17 storeman kernel: usb 5-2.1.2.4: device descriptor read/64, error -71
Feb 04 22:23:17 storeman kernel: usb 5-2.1.2.4: device descriptor read/64, error -71
45 
# journalctl -b1 | grep -i Bios
Specifying boot ID or boot offset has no effect, no persistent journal was found.
46

b1 means the last bootlog, no clue why you get this error

47

Separately, remember how copying from NVMe in Windows was slow? It’s fast again now (still not as fast as it should be):

Local NVMe to NVMe

image
image449×287 5.61 KB

image
image449×287 5.81 KB

Copying from NAS to NVMe

Still slow to copy over Ethernet though. That means the question about read speeds isn’t yet answered.

image
image449×287 6.21 KB

48

I don’t know, use an old HDD or what ever you have lying around and install Linux, if not
boot your PC from a Linux Live ISO.
My Connext -3 pro works with it, not sure with the CX-6, but should also work.
Use dd and read from dev/zero and write via SMB to your NAS.

49

write a file with dd and use it for fio to read

dd if=/dev/zero of=/mnt/store01/fio2.tmp bs=1G count=50

then

fio --filename=/mnt/store01/fio2.tmp --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
50

@Sawtaytoes can you print a zpool status ?

51

and “zfs get all Wolves”

52

Always have been, try Teracopy? TeraCopy for Windows - Code Sector

53

I can try Teracopy later to see if that fixes local Windows transfers.

I included zpool iostat -vl, since you’ll probably ask for that next :stuck_out_tongue:.

And I made sure to get the Test dataset as well since that’s what I’m using to test. It’s setup for SMB.

# zfs get all Wolves
NAME    PROPERTY              VALUE                  SOURCE
Wolves  type                  filesystem             -
Wolves  creation              Wed Dec 28 22:11 2022  -
Wolves  used                  15.4T                  -
Wolves  available             20.6T                  -
Wolves  referenced            128K                   -
Wolves  compressratio         1.04x                  -
Wolves  mounted               yes                    -
Wolves  quota                 none                   default
Wolves  reservation           none                   default
Wolves  recordsize            128K                   default
Wolves  mountpoint            /mnt/Wolves            default
Wolves  sharenfs              off                    local
Wolves  checksum              on                     default
Wolves  compression           lz4                    local
Wolves  atime                 off                    local
Wolves  devices               on                     default
Wolves  exec                  on                     default
Wolves  setuid                on                     default
Wolves  readonly              off                    default
Wolves  zoned                 off                    default
Wolves  snapdir               hidden                 default
Wolves  aclmode               discard                local
Wolves  aclinherit            discard                local
Wolves  createtxg             1                      -
Wolves  canmount              on                     default
Wolves  xattr                 on                     default
Wolves  copies                1                      default
Wolves  version               5                      -
Wolves  utf8only              off                    -
Wolves  normalization         none                   -
Wolves  casesensitivity       sensitive              -
Wolves  vscan                 off                    default
Wolves  nbmand                off                    default
Wolves  sharesmb              off                    local
Wolves  refquota              none                   default
Wolves  refreservation        none                   default
Wolves  guid                  17938368032450470770   -
Wolves  primarycache          all                    default
Wolves  secondarycache        all                    default
Wolves  usedbysnapshots       0B                     -
Wolves  usedbydataset         128K                   -
Wolves  usedbychildren        15.4T                  -
Wolves  usedbyrefreservation  0B                     -
Wolves  logbias               latency                default
Wolves  objsetid              54                     -
Wolves  dedup                 off                    default
Wolves  mlslabel              none                   default
Wolves  sync                  standard               default
Wolves  dnodesize             legacy                 default
Wolves  refcompressratio      1.00x                  -
Wolves  written               128K                   -
Wolves  logicalused           16.0T                  -
Wolves  logicalreferenced     54.5K                  -
Wolves  volmode               default                default
Wolves  filesystem_limit      none                   default
Wolves  snapshot_limit        none                   default
Wolves  filesystem_count      none                   default
Wolves  snapshot_count        none                   default
Wolves  snapdev               hidden                 default
Wolves  acltype               posix                  local
Wolves  context               none                   default
Wolves  fscontext             none                   default
Wolves  defcontext            none                   default
Wolves  rootcontext           none                   default
Wolves  relatime              off                    default
Wolves  redundant_metadata    all                    default
Wolves  overlay               on                     default
Wolves  encryption            off                    default
Wolves  keylocation           none                   default
Wolves  keyformat             none                   default
Wolves  pbkdf2iters           0                      default
Wolves  special_small_blocks  0                      default

# zfs get all Wolves/Test
NAME         PROPERTY               VALUE                  SOURCE
Wolves/Test  type                   filesystem             -
Wolves/Test  creation               Thu Feb  9 23:33 2023  -
Wolves/Test  used                   20.6G                  -
Wolves/Test  available              20.6T                  -
Wolves/Test  referenced             20.6G                  -
Wolves/Test  compressratio          1.00x                  -
Wolves/Test  mounted                yes                    -
Wolves/Test  quota                  none                   local
Wolves/Test  reservation            none                   local
Wolves/Test  recordsize             128K                   default
Wolves/Test  mountpoint             /mnt/Wolves/Test       default
Wolves/Test  sharenfs               off                    inherited from Wolves
Wolves/Test  checksum               on                     default
Wolves/Test  compression            lz4                    inherited from Wolves
Wolves/Test  atime                  off                    inherited from Wolves
Wolves/Test  devices                on                     default
Wolves/Test  exec                   on                     default
Wolves/Test  setuid                 on                     default
Wolves/Test  readonly               off                    default
Wolves/Test  zoned                  off                    default
Wolves/Test  snapdir                hidden                 default
Wolves/Test  aclmode                restricted             local
Wolves/Test  aclinherit             passthrough            local
Wolves/Test  createtxg              709356                 -
Wolves/Test  canmount               on                     default
Wolves/Test  xattr                  sa                     local
Wolves/Test  copies                 1                      local
Wolves/Test  version                5                      -
Wolves/Test  utf8only               off                    -
Wolves/Test  normalization          none                   -
Wolves/Test  casesensitivity        insensitive            -
Wolves/Test  vscan                  off                    default
Wolves/Test  nbmand                 off                    default
Wolves/Test  sharesmb               off                    inherited from Wolves
Wolves/Test  refquota               none                   local
Wolves/Test  refreservation         none                   local
Wolves/Test  guid                   10891789576475650459   -
Wolves/Test  primarycache           all                    default
Wolves/Test  secondarycache         all                    default
Wolves/Test  usedbysnapshots        0B                     -
Wolves/Test  usedbydataset          20.6G                  -
Wolves/Test  usedbychildren         0B                     -
Wolves/Test  usedbyrefreservation   0B                     -
Wolves/Test  logbias                latency                default
Wolves/Test  objsetid               15934                  -
Wolves/Test  dedup                  off                    default
Wolves/Test  mlslabel               none                   default
Wolves/Test  sync                   standard               default
Wolves/Test  dnodesize              legacy                 default
Wolves/Test  refcompressratio       1.00x                  -
Wolves/Test  written                20.6G                  -
Wolves/Test  logicalused            20.6G                  -
Wolves/Test  logicalreferenced      20.6G                  -
Wolves/Test  volmode                default                default
Wolves/Test  filesystem_limit       none                   default
Wolves/Test  snapshot_limit         none                   default
Wolves/Test  filesystem_count       none                   default
Wolves/Test  snapshot_count         none                   default
Wolves/Test  snapdev                hidden                 default
Wolves/Test  acltype                nfsv4                  local
Wolves/Test  context                none                   default
Wolves/Test  fscontext              none                   default
Wolves/Test  defcontext             none                   default
Wolves/Test  rootcontext            none                   default
Wolves/Test  relatime               off                    default
Wolves/Test  redundant_metadata     all                    default
Wolves/Test  overlay                on                     default
Wolves/Test  encryption             off                    default
Wolves/Test  keylocation            none                   default
Wolves/Test  keyformat              none                   default
Wolves/Test  pbkdf2iters            0                      default
Wolves/Test  special_small_blocks   0                      default
Wolves/Test  org.truenas:managedby  10.1.0.247             local

# zpool status
  pool: Bunnies
 state: ONLINE
  scan: resilvered 1.13T in 01:00:15 with 0 errors on Mon Jan 23 21:51:02 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        Bunnies                                   ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            d07c0740-7e13-11ed-a976-a8a159c2849a  ONLINE       0     0     0
            d0a1665c-7e13-11ed-a976-a8a159c2849a  ONLINE       0     0     0
          mirror-1                                ONLINE       0     0     0
            d0a69595-7e13-11ed-a976-a8a159c2849a  ONLINE       0     0     0
            d0ac68be-7e13-11ed-a976-a8a159c2849a  ONLINE       0     0     0
          mirror-2                                ONLINE       0     0     0
            d0b4495b-7e13-11ed-a976-a8a159c2849a  ONLINE       0     0     0
            d0ba8170-7e13-11ed-a976-a8a159c2849a  ONLINE       0     0     0
          mirror-3                                ONLINE       0     0     0
            fafb2e96-d1a5-45a5-8fa4-c79cbe9439dd  ONLINE       0     0     0
            b9790124-e168-49d7-bf15-4ef5d92c9727  ONLINE       0     0     0
          mirror-4                                ONLINE       0     0     0
            81d41751-fe27-4b1a-a45f-9ec11efd0ec2  ONLINE       0     0     0
            da8385bc-ce10-43f4-b1d6-b75ac22fc886  ONLINE       0     0     0
          mirror-5                                ONLINE       0     0     0
            070917b1-2043-4c93-8c46-34c6ec1f0a8a  ONLINE       0     0     0
            26bad8a4-7d6f-4542-8c17-d1c5aec98440  ONLINE       0     0     0
          mirror-6                                ONLINE       0     0     0
            d10956de-72eb-441f-864f-b1b810e3ed37  ONLINE       0     0     0
            46d58c6f-cf6c-4e87-80b4-0382b642a472  ONLINE       0     0     0
          mirror-7                                ONLINE       0     0     0
            a6620c29-00ef-491e-a76f-21a9cfb0d5fb  ONLINE       0     0     0
            12288a27-cd2a-48b7-923c-cc5897a10ee1  ONLINE       0     0     0
          mirror-8                                ONLINE       0     0     0
            fe6db0d0-45ca-4a9e-96d3-0c6d9ff235c3  ONLINE       0     0     0
            e6e0c7e6-b380-40ca-87e1-25292cad6215  ONLINE       0     0     0
          mirror-9                                ONLINE       0     0     0
            0fb792da-4f62-4b7b-8e7c-734e185ce37b  ONLINE       0     0     0
            485016cb-069a-4c4b-be78-05f472396179  ONLINE       0     0     0
          mirror-11                               ONLINE       0     0     0
            aad8d68e-1763-4c44-aa42-d5b95521bc12  ONLINE       0     0     0
            0c34ca9a-787d-4486-bab1-49edffee1432  ONLINE       0     0     0
          mirror-15                               ONLINE       0     0     0
            d8569602-18d1-441e-a97b-0c15c73c5776  ONLINE       0     0     0
            e7167f98-c5c0-4a5d-bda2-4348579bf98d  ONLINE       0     0     0
          mirror-16                               ONLINE       0     0     0
            dbc073ca-54e6-4391-8086-95f61a7d97bd  ONLINE       0     0     0
            1771fac6-ce5e-4ed9-afb1-cd840cad7cf4  ONLINE       0     0     0
          mirror-17                               ONLINE       0     0     0
            c2ef0edd-8450-4874-a238-8a89c8ef0f90  ONLINE       0     0     0
            a30ec5fa-b252-48c3-87f1-187146911fe5  ONLINE       0     0     0
          mirror-18                               ONLINE       0     0     0
            83cddd5d-5778-4a87-890d-885134f5d897  ONLINE       0     0     0
            db183522-817f-489e-b1bd-078867d52cd6  ONLINE       0     0     0
          mirror-19                               ONLINE       0     0     0
            ba02825f-88df-49a6-96c8-347a25a4ea6e  ONLINE       0     0     0
            fe16ba6b-22a7-48fd-b2d6-81bbb5e1c22f  ONLINE       0     0     0
          mirror-20                               ONLINE       0     0     0
            d0548c12-6f97-42b4-986d-5f53cb79d682  ONLINE       0     0     0
            f4394834-efa9-45e3-8553-133c3e73091f  ONLINE       0     0     0
        dedup
          mirror-10                               ONLINE       0     0     0
            d3165df5-65b9-4e10-ab8a-07dd31c9964d  ONLINE       0     0     0
            4072593c-3a2b-4f20-a67a-2cc127e0c3b0  ONLINE       0     0     0
        special
          mirror-13                               ONLINE       0     0     0
            4f961158-125c-474c-a90b-fdc6f9833f4a  ONLINE       0     0     0
            ac5c6fe2-0b3b-4941-9337-28649600136e  ONLINE       0     0     0
          mirror-14                               ONLINE       0     0     0
            25e755ed-cc1c-438c-890c-d9ff5be64191  ONLINE       0     0     0
            548a3fcd-4aba-4b70-814c-079e388e41dc  ONLINE       0     0     0
        logs
          mirror-12                               ONLINE       0     0     0
            f9d7758f-96ad-423c-a73f-191c67267883  ONLINE       0     0     0
            711d8414-4c93-4dac-9cc5-2ce9f4cb859c  ONLINE       0     0     0
        cache
          cad0fad3-fddb-4c0f-8367-6f088c8feaf8    ONLINE       0     0     0
          ebc2cfbc-3d38-4be5-a7c8-6efd4a0ddce2    ONLINE       0     0     0
        spares
          2134b3a6-8e6a-496a-9ffa-83a3a4c28c7b    AVAIL   
          d7cbdb23-5aee-4f98-9817-be2a639d9387    AVAIL   

errors: No known data errors

  pool: TrueNAS-Apps
 state: ONLINE
  scan: scrub repaired 0B in 00:02:02 with 0 errors on Mon Jan 16 17:26:56 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        TrueNAS-Apps                              ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            53c7f825-cb94-4237-8c79-728becb6c753  ONLINE       0     0     0
            b531e1a4-3069-4c90-9437-77eeb333eaac  ONLINE       0     0     0

errors: No known data errors

  pool: Wolves
 state: ONLINE
  scan: resilvered 27.1G in 00:01:03 with 0 errors on Wed Jan 25 01:33:44 2023
config:

        NAME                                      STATE     READ WRITE CKSUM
        Wolves                                    ONLINE       0     0     0
          mirror-0                                ONLINE       0     0     0
            5dfaf5ee-a57f-44cb-acf6-b97130bc2ee1  ONLINE       0     0     0
            2da54a3c-8725-478e-adaf-1a7a53c0890b  ONLINE       0     0     0
          mirror-1                                ONLINE       0     0     0
            2bbaffd9-ac67-4239-b3d7-81046dc4bd24  ONLINE       0     0     0
            e631eb85-1ab7-40b5-8161-727e8d1a9a3a  ONLINE       0     0     0
          mirror-2                                ONLINE       0     0     0
            f287f68c-2293-4d4b-abde-e6e019faf73b  ONLINE       0     0     0
            e471104b-c2ab-4e2b-9c7c-14aa187aef32  ONLINE       0     0     0
          mirror-3                                ONLINE       0     0     0
            33dcaafb-6eae-4931-8743-20f9b32d44fd  ONLINE       0     0     0
            4fa1a66c-effc-4531-b5c0-8c514293055d  ONLINE       0     0     0
          mirror-4                                ONLINE       0     0     0
            d2edc92c-4250-40c4-b382-0beb39ab8d6d  ONLINE       0     0     0
            0da118f2-7df8-42ee-8a94-fa02dcca1b96  ONLINE       0     0     0
          mirror-5                                ONLINE       0     0     0
            11fcc837-6d7c-466e-b2c0-a0d5e8c82fa5  ONLINE       0     0     0
            435d8322-9143-42fe-803b-7b6abbe75baa  ONLINE       0     0     0
          mirror-6                                ONLINE       0     0     0
            dc9cafbd-4c82-4ae0-a3be-b47ed18fa6ac  ONLINE       0     0     0
            3bb29f56-8158-4ac9-8d25-8aff78bd1f7f  ONLINE       0     0     0
          mirror-7                                ONLINE       0     0     0
            6084d12e-e15c-4f89-8bd6-396bc723fb84  ONLINE       0     0     0
            72fa67ef-c81d-4b98-a7b7-aa192e1625ea  ONLINE       0     0     0
          mirror-8                                ONLINE       0     0     0
            e8bc4881-36f2-45f3-9795-597ce62e0e05  ONLINE       0     0     0
            2b55a8d1-a442-45b8-8431-853ca7b10e9b  ONLINE       0     0     0
          mirror-9                                ONLINE       0     0     0
            0c8790ab-d731-4ae2-8b4d-f134b02929d2  ONLINE       0     0     0
            ec03bf0d-4168-418f-b83b-7e21e457ca83  ONLINE       0     0     0
        spares
          7ea93cff-4798-4451-8914-8bd8475340b3    AVAIL   
          a9e3750e-7381-436d-a4de-8d68a8136bdd    AVAIL   

errors: No known data errors

  pool: boot-pool
 state: ONLINE
  scan: scrub repaired 0B in 00:00:11 with 0 errors on Tue Feb 14 03:45:12 2023
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            sdbi3   ONLINE       0     0     0
            sdbj3   ONLINE       0     0     0

errors: No known data errors

# zpool iostat -vl
                                            capacity     operations     bandwidth    total_wait     disk_wait    syncq_wait    asyncq_wait  scrub   trim
pool                                      alloc   free   read  write   read  write   read  write   read  write   read  write   read  write   wait   wait
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
Bunnies                                   16.7T  17.7T     33     27  3.99M  2.70M    1ms   10ms    1ms  499us   18us  946ns  112us   10ms  769ns  488us
  mirror-0                                1.19T   636G      1      0   234K   113K    1ms   16ms    1ms  808us  347us  657ns  107us   16ms      -      -
    d07c0740-7e13-11ed-a976-a8a159c2849a      -      -      0      0   117K  56.7K    1ms   15ms    1ms  769us   18us  689ns   97us   15ms      -      -
    d0a1665c-7e13-11ed-a976-a8a159c2849a      -      -      0      0   117K  56.7K    1ms   17ms    1ms  847us  675us  624ns  117us   17ms      -      -
  mirror-1                                1.19T   636G      1      0   226K   113K    1ms   16ms    1ms  807us  264us  667ns   95us   16ms      -      -
    d0a69595-7e13-11ed-a976-a8a159c2849a      -      -      0      0   113K  56.5K    1ms   17ms    1ms  843us  485us  713ns  108us   17ms      -      -
    d0ac68be-7e13-11ed-a976-a8a159c2849a      -      -      0      0   113K  56.5K    1ms   16ms    1ms  771us   48us  620ns   82us   15ms      -      -
  mirror-2                                1.19T   639G      1      0   230K   114K    1ms   16ms    1ms  787us   26us  660ns   89us   15ms      -      -
    d0b4495b-7e13-11ed-a976-a8a159c2849a      -      -      0      0   115K  57.1K    1ms   15ms    1ms  740us   16us  698ns   78us   14ms      -      -
    d0ba8170-7e13-11ed-a976-a8a159c2849a      -      -      0      0   115K  57.1K    2ms   17ms    1ms  833us   35us  623ns  101us   16ms      -      -
  mirror-3                                1.22T   611G      1      0   238K   114K    1ms   15ms    1ms  748us   44us  659ns   68us   15ms      -      -
    fafb2e96-d1a5-45a5-8fa4-c79cbe9439dd      -      -      0      0   119K  57.1K    1ms   15ms    1ms  770us   86us  702ns   80us   15ms      -      -
    b9790124-e168-49d7-bf15-4ef5d92c9727      -      -      0      0   118K  57.1K    1ms   15ms    1ms  725us    1us  616ns   56us   15ms      -      -
  mirror-4                                1.22T   606G      1      0   238K   114K    1ms   15ms    1ms  729us    1us  641ns   60us   15ms      -      -
    81d41751-fe27-4b1a-a45f-9ec11efd0ec2      -      -      0      0   118K  57.0K    1ms   15ms    1ms  730us    2us  678ns   58us   15ms      -      -
    da8385bc-ce10-43f4-b1d6-b75ac22fc886      -      -      0      0   119K  57.0K    1ms   15ms    1ms  727us    1us  604ns   62us   15ms      -      -
  mirror-5                                1.22T   609G      1      0   236K   114K    1ms   15ms    1ms  722us    8us  652ns   53us   15ms      -      -
    070917b1-2043-4c93-8c46-34c6ec1f0a8a      -      -      0      0   118K  57.2K    1ms   15ms    1ms  722us   14us  676ns   53us   15ms      -      -
    26bad8a4-7d6f-4542-8c17-d1c5aec98440      -      -      0      0   118K  57.2K    1ms   15ms    1ms  722us    2us  628ns   54us   15ms      -      -
  mirror-6                                1.22T   607G      1      0   237K   114K    1ms   15ms    1ms  734us  115us  671ns   71us   15ms      -      -
    d10956de-72eb-441f-864f-b1b810e3ed37      -      -      0      0   119K  57.0K    1ms   15ms    1ms  733us  161us  710ns   69us   15ms      -      -
    46d58c6f-cf6c-4e87-80b4-0382b642a472      -      -      0      0   118K  57.0K    1ms   15ms    1ms  736us   69us  632ns   72us   15ms      -      -
  mirror-7                                1.22T   609G      1      0   236K   114K    1ms   15ms    1ms  730us    2us  658ns   60us   15ms      -      -
    a6620c29-00ef-491e-a76f-21a9cfb0d5fb      -      -      0      0   118K  57.1K    1ms   15ms    1ms  729us    2us  690ns   59us   14ms      -      -
    12288a27-cd2a-48b7-923c-cc5897a10ee1      -      -      0      0   118K  57.1K    1ms   15ms    1ms  730us    1us  626ns   61us   15ms      -      -
  mirror-8                                1.22T   610G      1      0   238K   115K    1ms   15ms    1ms  723us    5us  665ns   62us   14ms      -      -
    fe6db0d0-45ca-4a9e-96d3-0c6d9ff235c3      -      -      0      0   119K  57.4K    1ms   15ms    1ms  725us    4us  697ns   62us   14ms      -      -
    e6e0c7e6-b380-40ca-87e1-25292cad6215      -      -      0      0   119K  57.4K    1ms   15ms    1ms  721us    7us  632ns   62us   14ms      -      -
  mirror-9                                1.22T   607G      1      0   237K   115K    1ms   15ms    1ms  720us    2us  653ns   57us   14ms      -      -
    0fb792da-4f62-4b7b-8e7c-734e185ce37b      -      -      0      0   119K  57.4K    1ms   15ms    1ms  720us    1us  684ns   58us   14ms      -      -
    485016cb-069a-4c4b-be78-05f472396179      -      -      0      0   119K  57.4K    1ms   15ms    1ms  720us    4us  621ns   56us   14ms      -      -
  mirror-11                               1.22T   607G      1      0   239K   115K    1ms   15ms    1ms  727us    3us  678ns   70us   14ms      -      -
    aad8d68e-1763-4c44-aa42-d5b95521bc12      -      -      0      0   119K  57.4K    1ms   15ms    1ms  728us    2us  710ns   68us   14ms      -      -
    0c34ca9a-787d-4486-bab1-49edffee1432      -      -      0      0   120K  57.4K    1ms   15ms    1ms  725us    4us  646ns   73us   14ms      -      -
  mirror-15                                657G  1.17T      2      1   289K   156K    1ms   16ms    1ms  686us   34us  643ns  194us   16ms      -      -
    d8569602-18d1-441e-a97b-0c15c73c5776      -      -      1      0   143K  77.8K    1ms   16ms    1ms  673us   10us  662ns  165us   16ms      -      -
    e7167f98-c5c0-4a5d-bda2-4348579bf98d      -      -      1      0   145K  77.8K    1ms   16ms    1ms  699us   58us  624ns  222us   16ms      -      -
  mirror-16                                658G  1.17T      2      1   286K   154K    1ms   15ms    1ms  669us   30us  660ns  174us   15ms      -      -
    dbc073ca-54e6-4391-8086-95f61a7d97bd      -      -      1      0   143K  76.9K    1ms   15ms    1ms  667us   19us  699ns  175us   15ms      -      -
    1771fac6-ce5e-4ed9-afb1-cd840cad7cf4      -      -      1      0   143K  76.9K    1ms   15ms    1ms  671us   42us  621ns  174us   15ms      -      -
  mirror-17                                650G  1.18T      2      1   284K   155K    1ms   15ms    1ms  667us   11us  665ns  172us   15ms      -      -
    c2ef0edd-8450-4874-a238-8a89c8ef0f90      -      -      1      0   142K  77.4K    1ms   15ms    1ms  667us    8us  693ns  172us   15ms      -      -
    a30ec5fa-b252-48c3-87f1-187146911fe5      -      -      1      0   142K  77.4K    1ms   15ms    1ms  667us   14us  638ns  172us   15ms      -      -
  mirror-18                                650G  1.18T      2      1   282K   154K    1ms   15ms    1ms  671us   23us  649ns  173us   15ms      -      -
    83cddd5d-5778-4a87-890d-885134f5d897      -      -      1      0   141K  77.0K    1ms   15ms    1ms  672us   13us  681ns  174us   15ms      -      -
    db183522-817f-489e-b1bd-078867d52cd6      -      -      1      0   141K  77.0K    1ms   15ms    1ms  671us   33us  617ns  172us   15ms      -      -
  mirror-19                                323G  1.50T      1      1   175K   172K    1ms   17ms    1ms  657us   51us  659ns  226us   16ms      -      -
    ba02825f-88df-49a6-96c8-347a25a4ea6e      -      -      0      0  87.5K  86.1K    1ms   16ms    1ms  655us   70us  704ns  220us   16ms      -      -
    fe16ba6b-22a7-48fd-b2d6-81bbb5e1c22f      -      -      0      0  87.5K  86.1K    1ms   17ms    1ms  659us   32us  614ns  233us   16ms      -      -
  mirror-20                                321G  1.50T      1      1   173K   171K    1ms   16ms    1ms  658us   43us  653ns  211us   15ms      -      -
    d0548c12-6f97-42b4-986d-5f53cb79d682      -      -      0      0  86.5K  85.4K    1ms   16ms    1ms  660us   61us  694ns  211us   15ms      -      -
    f4394834-efa9-45e3-8553-133c3e73091f      -      -      0      0  86.1K  85.4K    1ms   16ms    1ms  657us   25us  612ns  212us   15ms      -      -
dedup                                         -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
  mirror-10                                   0  1.86T      0      0      1     91   28ms   55us   28ms   53us  685ns  706ns      -      -      -      -
    d3165df5-65b9-4e10-ab8a-07dd31c9964d      -      -      0      0      0     45   28ms   57us   28ms   56us  822ns  729ns      -      -      -      -
    4072593c-3a2b-4f20-a67a-2cc127e0c3b0      -      -      0      0      0     45   28ms   52us   28ms   51us  548ns  683ns      -      -      -      -
special                                       -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
  mirror-13                                102G   786G      0      2  2.66K  32.8K   27us  117us   26us   50us  897ns  638ns    1us   67us  940ns  494us
    4f961158-125c-474c-a90b-fdc6f9833f4a      -      -      0      1  1.33K  16.4K   27us  116us   26us   51us  892ns  684ns    1us   66us    1us  471us
    ac5c6fe2-0b3b-4941-9337-28649600136e      -      -      0      1  1.33K  16.4K   27us  118us   25us   50us  902ns  591ns    1us   68us  626ns  517us
  mirror-14                                102G   786G      0      2  2.67K  37.5K   27us  125us   25us   48us  974ns  603ns    1us   77us  664ns  485us
    25e755ed-cc1c-438c-890c-d9ff5be64191      -      -      0      1  1.34K  18.8K   27us  124us   25us   48us  978ns  629ns    1us   76us  785ns  463us
    548a3fcd-4aba-4b70-814c-079e388e41dc      -      -      0      1  1.33K  18.8K   27us  125us   25us   48us  969ns  577ns    1us   78us  543ns  507us
logs                                          -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
  mirror-12                                540K   888G      0      3      1   474K    8ms    1ms    8ms  112us  768ns    2us      -    1ms      -  367us
    f9d7758f-96ad-423c-a73f-191c67267883      -      -      0      1      0   237K    2ms    1ms    2ms  112us  822ns    2us      -    1ms      -  313us
    711d8414-4c93-4dac-9cc5-2ce9f4cb859c      -      -      0      1      0   237K   14ms    1ms   14ms  112us  713ns    1us      -    1ms      -  421us
cache                                         -      -      -      -      -      -      -      -      -      -      -      -      -      -      -      -
  cad0fad3-fddb-4c0f-8367-6f088c8feaf8     879G  15.7G      1      2  12.3K   306K   29us    4ms   28us  188us  934ns    1us  840ns    4ms      -      -
  ebc2cfbc-3d38-4be5-a7c8-6efd4a0ddce2     879G  15.6G      1      2  12.6K   305K   29us    4ms   28us  187us  926ns  768ns  892ns    4ms      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
TrueNAS-Apps                              38.8G   423G      0     41  13.5K   896K    2ms    1ms  434us  173us    3us    4ms    4ms    1ms   24ms    4ms
  mirror-0                                38.8G   423G      0     41  13.5K   896K    2ms    1ms  434us  173us    3us    4ms    4ms    1ms   24ms    4ms
    53c7f825-cb94-4237-8c79-728becb6c753      -      -      0     20  6.75K   448K    2ms    1ms  431us  193us    1us    4ms    4ms    1ms   42ms    4ms
    b531e1a4-3069-4c90-9437-77eeb333eaac      -      -      0     21  6.72K   448K    2ms    1ms  438us  155us    5us    4ms    4ms    1ms   11ms    4ms
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
Wolves                                    15.4T  20.9T      0     38  19.2K  4.21M  906us   17ms  703us    1ms    1us    4us  374us   17ms   19ms      -
  mirror-0                                2.77T   871G      0      3  1.86K   360K    1ms   17ms  869us    1ms  835ns    5us    1ms   16ms    2ms      -
    5dfaf5ee-a57f-44cb-acf6-b97130bc2ee1      -      -      0      1  1.00K   180K    1ms   17ms  842us    1ms  837ns  700ns    1ms   16ms    2ms      -
    2da54a3c-8725-478e-adaf-1a7a53c0890b      -      -      0      1    879   180K    1ms   17ms  898us    1ms  833ns   11us    1ms   16ms    2ms      -
  mirror-1                                2.85T   798G      0      2  1.93K   342K    2ms   17ms  825us    1ms    9us    2us    2ms   16ms      -      -
    2bbaffd9-ac67-4239-b3d7-81046dc4bd24      -      -      0      1    961   171K    2ms   17ms  871us    1ms   18us    3us    3ms   16ms      -      -
    e631eb85-1ab7-40b5-8161-727e8d1a9a3a      -      -      0      1   1014   171K    1ms   17ms  780us    1ms  812ns  619ns    1ms   16ms      -      -
  mirror-2                                2.88T   767G      0      2  1.92K   343K  665us   17ms  660us    1ms  837ns    5us    4us   16ms    2ms      -
    f287f68c-2293-4d4b-abde-e6e019faf73b      -      -      0      1    952   172K  674us   17ms  670us    1ms  831ns  715ns    4us   16ms    2ms      -
    e471104b-c2ab-4e2b-9c7c-14aa187aef32      -      -      0      1   1017   172K  656us   17ms  651us    1ms  843ns   10us    4us   16ms    2ms      -
  mirror-3                                1.23T  2.39T      0      4  1.89K   452K  717us   18ms  694us    1ms  997ns  661ns   12us   18ms   19ms      -
    33dcaafb-6eae-4931-8743-20f9b32d44fd      -      -      0      2  1.02K   226K  723us   18ms  696us    1ms  884ns  705ns    7us   17ms   32ms      -
    4fa1a66c-effc-4531-b5c0-8c514293055d      -      -      0      2    884   226K  709us   19ms  692us    1ms    1us  616ns   18us   18ms    7ms      -
  mirror-4                                1.27T  2.36T      0      4  1.99K   467K  669us   17ms  614us    1ms  859ns    4us    5us   16ms   42ms      -
    d2edc92c-4250-40c4-b382-0beb39ab8d6d      -      -      0      2  1.04K   233K  622us   17ms  595us    1ms  863ns  703ns    4us   16ms   18ms      -
    0da118f2-7df8-42ee-8a94-fa02dcca1b96      -      -      0      2    972   233K  720us   17ms  633us    1ms  855ns    8us    6us   16ms   68ms      -
  mirror-5                                1.26T  2.37T      0      4  1.99K   469K  621us   16ms  610us    1ms  860ns    5us   10us   15ms    6ms      -
    11fcc837-6d7c-466e-b2c0-a0d5e8c82fa5      -      -      0      2  1.07K   234K  585us   16ms  575us    1ms  852ns  696ns    6us   15ms    7ms      -
    435d8322-9143-42fe-803b-7b6abbe75baa      -      -      0      2    943   234K  661us   16ms  649us    1ms  868ns   10us   16us   15ms    5ms      -
  mirror-6                                1.26T  2.36T      0      4  1.99K   467K  607us   17ms  603us    1ms  846ns    6us    6us   16ms    2ms      -
    dc9cafbd-4c82-4ae0-a3be-b47ed18fa6ac      -      -      0      2  1.02K   234K  606us   17ms  602us    1ms  846ns   11us    7us   16ms    2ms      -
    3bb29f56-8158-4ac9-8d25-8aff78bd1f7f      -      -      0      2    989   234K  608us   17ms  605us    1ms  846ns  613ns    4us   16ms    2ms      -
  mirror-7                                1.25T  2.37T      0      3  2.00K   445K  679us   18ms  670us    1ms  862ns  655ns   17us   17ms      -      -
    6084d12e-e15c-4f89-8bd6-396bc723fb84      -      -      0      1    918   223K  716us   17ms  700us    1ms  874ns  701ns   30us   16ms      -      -
    72fa67ef-c81d-4b98-a7b7-aa192e1625ea      -      -      0      1  1.10K   223K  647us   20ms  643us    1ms  850ns  610ns    6us   19ms      -      -
  mirror-8                                 322G  3.31T      0      4  1.65K   476K  986us   18ms  852us    1ms  925ns    4us  285us   18ms    1ms      -
    e8bc4881-36f2-45f3-9795-597ce62e0e05      -      -      0      2    873   238K  932us   18ms  809us    1ms  930ns    9us  279us   18ms  105us      -
    2b55a8d1-a442-45b8-8431-853ca7b10e9b      -      -      0      2    815   238K    1ms   18ms  897us    1ms  919ns  620ns  293us   18ms    4ms      -
  mirror-9                                 312G  3.32T      0      4  1.99K   487K  705us   19ms  698us    1ms  910ns    5us   12us   19ms      -      -
    0c8790ab-d731-4ae2-8b4d-f134b02929d2      -      -      0      2   1018   243K  720us   19ms  714us    1ms  909ns  705ns   10us   19ms      -      -
    ec03bf0d-4168-418f-b83b-7e21e457ca83      -      -      0      2   1022   243K  690us   19ms  682us    1ms  912ns   10us   14us   19ms      -      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
boot-pool                                 2.98G  52.0G      0     40  15.4K   896K    9ms    2ms  590us  244us    2us    1us   40us    2ms   15ms      -
  mirror-0                                2.98G  52.0G      0     40  15.4K   896K    9ms    2ms  590us  244us    2us    1us   40us    2ms   15ms      -
    sdbi3                                     -      -      0     20  7.73K   448K   11ms    2ms  599us  247us    2us    1us   33us    2ms   18ms      -
    sdbj3                                     -      -      0     20  7.71K   448K    8ms    2ms  582us  241us    2us    1us   47us    2ms   13ms      -
----------------------------------------  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----  -----
54

Wolves

# dd if=/dev/zero of=/mnt/Wolves/Test/fio3.tmp bs=1G count=50
50+0 records in
50+0 records out
53687091200 bytes (54 GB, 50 GiB) copied, 22.0829 s, 2.4 GB/s

Here we go! This is much better, but like you said, it’s probably memory.

The most these drives can theoretically do is 600MB/s. So with 20 of them, that’s 12GB/s.

# fio --filename=/mnt/Wolves/Test/fio3.tmp --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=10.5GiB/s][r=172k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=10.0GiB/s][r=164k IOPS][eta 01m:55s] 
Jobs: 4 (f=4): [R(4)][5.8%][r=10.1GiB/s][r=166k IOPS][eta 01m:53s] 
Jobs: 4 (f=4): [R(4)][7.5%][r=10.2GiB/s][r=166k IOPS][eta 01m:51s] 
Jobs: 4 (f=4): [R(4)][9.2%][r=9.96GiB/s][r=163k IOPS][eta 01m:49s] 
Jobs: 4 (f=4): [R(4)][10.9%][r=9.00GiB/s][r=164k IOPS][eta 01m:46s] 
Jobs: 4 (f=4): [R(4)][12.5%][r=10.0GiB/s][r=164k IOPS][eta 01m:45s] 
Jobs: 4 (f=4): [R(4)][14.2%][r=9.98GiB/s][r=164k IOPS][eta 01m:43s] 
Jobs: 4 (f=4): [R(4)][15.8%][r=10.0GiB/s][r=165k IOPS][eta 01m:41s] 
Jobs: 4 (f=4): [R(4)][17.5%][r=9.96GiB/s][r=163k IOPS][eta 01m:39s] 
Jobs: 4 (f=4): [R(4)][19.2%][r=10.1GiB/s][r=165k IOPS][eta 01m:37s] 
Jobs: 4 (f=4): [R(4)][20.8%][r=10.0GiB/s][r=164k IOPS][eta 01m:35s] 
Jobs: 4 (f=4): [R(4)][22.5%][r=10.0GiB/s][r=165k IOPS][eta 01m:33s] 
Jobs: 4 (f=4): [R(4)][24.2%][r=10.0GiB/s][r=164k IOPS][eta 01m:31s] 
Jobs: 4 (f=4): [R(4)][26.1%][r=10.0GiB/s][r=164k IOPS][eta 01m:28s] 
Jobs: 4 (f=4): [R(4)][27.5%][r=9.89GiB/s][r=162k IOPS][eta 01m:27s] 
Jobs: 4 (f=4): [R(4)][29.2%][r=10.1GiB/s][r=165k IOPS][eta 01m:25s] 
Jobs: 4 (f=4): [R(4)][31.1%][r=10.0GiB/s][r=164k IOPS][eta 01m:22s] 
Jobs: 4 (f=4): [R(4)][32.8%][r=10.0GiB/s][r=165k IOPS][eta 01m:20s] 
Jobs: 4 (f=4): [R(4)][34.2%][r=9.97GiB/s][r=163k IOPS][eta 01m:19s] 
Jobs: 4 (f=4): [R(4)][35.8%][r=10.0GiB/s][r=164k IOPS][eta 01m:17s] 
Jobs: 4 (f=4): [R(4)][37.5%][r=10.1GiB/s][r=166k IOPS][eta 01m:15s] 
Jobs: 4 (f=4): [R(4)][39.2%][r=10.0GiB/s][r=164k IOPS][eta 01m:13s] 
Jobs: 4 (f=4): [R(4)][40.8%][r=10.1GiB/s][r=165k IOPS][eta 01m:11s] 
Jobs: 4 (f=4): [R(4)][42.5%][r=10.1GiB/s][r=165k IOPS][eta 01m:09s] 
Jobs: 4 (f=4): [R(4)][44.2%][r=9.86GiB/s][r=161k IOPS][eta 01m:07s] 
Jobs: 4 (f=4): [R(4)][45.8%][r=9.93GiB/s][r=163k IOPS][eta 01m:05s] 
Jobs: 4 (f=4): [R(4)][47.5%][r=9.97GiB/s][r=163k IOPS][eta 01m:03s] 
Jobs: 4 (f=4): [R(4)][49.2%][r=10.1GiB/s][r=165k IOPS][eta 01m:01s] 
Jobs: 4 (f=4): [R(4)][50.8%][r=10.1GiB/s][r=165k IOPS][eta 00m:59s] 
Jobs: 4 (f=4): [R(4)][52.5%][r=9.97GiB/s][r=163k IOPS][eta 00m:57s] 
Jobs: 4 (f=4): [R(4)][54.2%][r=9.93GiB/s][r=163k IOPS][eta 00m:55s] 
Jobs: 4 (f=4): [R(4)][55.8%][r=9.87GiB/s][r=162k IOPS][eta 00m:53s] 
Jobs: 4 (f=4): [R(4)][57.5%][r=10.3GiB/s][r=169k IOPS][eta 00m:51s] 
Jobs: 4 (f=4): [R(4)][59.2%][r=10.4GiB/s][r=170k IOPS][eta 00m:49s] 
Jobs: 4 (f=4): [R(4)][60.8%][r=10.6GiB/s][r=173k IOPS][eta 00m:47s] 
Jobs: 4 (f=4): [R(4)][62.5%][r=10.8GiB/s][r=177k IOPS][eta 00m:45s] 
Jobs: 4 (f=4): [R(4)][64.7%][r=9.95GiB/s][r=163k IOPS][eta 00m:42s] 
Jobs: 4 (f=4): [R(4)][65.8%][r=9.95GiB/s][r=163k IOPS][eta 00m:41s] 
Jobs: 4 (f=4): [R(4)][67.5%][r=10.1GiB/s][r=165k IOPS][eta 00m:39s] 
Jobs: 4 (f=4): [R(4)][69.2%][r=9.97GiB/s][r=163k IOPS][eta 00m:37s] 
Jobs: 4 (f=4): [R(4)][71.4%][r=10.0GiB/s][r=164k IOPS][eta 00m:34s] 
Jobs: 4 (f=4): [R(4)][72.5%][r=10.1GiB/s][r=166k IOPS][eta 00m:33s] 
Jobs: 4 (f=4): [R(4)][74.2%][r=10.0GiB/s][r=164k IOPS][eta 00m:31s] 
Jobs: 4 (f=4): [R(4)][75.8%][r=9.98GiB/s][r=163k IOPS][eta 00m:29s] 
Jobs: 4 (f=4): [R(4)][77.5%][r=9.89GiB/s][r=162k IOPS][eta 00m:27s] 
Jobs: 4 (f=4): [R(4)][79.2%][r=10.1GiB/s][r=165k IOPS][eta 00m:25s] 
Jobs: 4 (f=4): [R(4)][80.8%][r=10.1GiB/s][r=165k IOPS][eta 00m:23s] 
Jobs: 4 (f=4): [R(4)][82.5%][r=10.1GiB/s][r=165k IOPS][eta 00m:21s] 
Jobs: 4 (f=4): [R(4)][84.2%][r=10.1GiB/s][r=165k IOPS][eta 00m:19s] 
Jobs: 4 (f=4): [R(4)][85.8%][r=9.99GiB/s][r=164k IOPS][eta 00m:17s] 
Jobs: 4 (f=4): [R(4)][87.5%][r=10.1GiB/s][r=165k IOPS][eta 00m:15s] 
Jobs: 4 (f=4): [R(4)][89.9%][r=9.99GiB/s][r=164k IOPS][eta 00m:12s] 
Jobs: 4 (f=4): [R(4)][90.8%][r=10.1GiB/s][r=166k IOPS][eta 00m:11s] 
Jobs: 4 (f=4): [R(4)][93.3%][r=10.1GiB/s][r=166k IOPS][eta 00m:08s] 
Jobs: 4 (f=4): [R(4)][94.2%][r=10.0GiB/s][r=164k IOPS][eta 00m:07s] 
Jobs: 4 (f=4): [R(4)][95.8%][r=10.1GiB/s][r=165k IOPS][eta 00m:05s] 
Jobs: 4 (f=4): [R(4)][97.5%][r=10.1GiB/s][r=166k IOPS][eta 00m:03s] 
Jobs: 4 (f=4): [R(4)][99.2%][r=10.1GiB/s][r=165k IOPS][eta 00m:01s] 
Jobs: 4 (f=4): [R(4)][100.0%][r=10.2GiB/s][r=166k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=2979723: Wed Feb 15 13:53:29 2023
  read: IOPS=165k, BW=10.1GiB/s (10.8GB/s)(1209GiB/120001msec)
    slat (usec): min=5, max=14170, avg=23.30, stdev= 5.71
    clat (nsec): min=1210, max=16482k, avg=1527114.41, stdev=91723.56
     lat (usec): min=21, max=16514, avg=1550.51, stdev=92.88
    clat percentiles (usec):
     |  1.00th=[ 1270],  5.00th=[ 1434], 10.00th=[ 1467], 20.00th=[ 1483],
     | 30.00th=[ 1500], 40.00th=[ 1516], 50.00th=[ 1516], 60.00th=[ 1532],
     | 70.00th=[ 1549], 80.00th=[ 1565], 90.00th=[ 1598], 95.00th=[ 1631],
     | 99.00th=[ 1795], 99.50th=[ 1958], 99.90th=[ 2212], 99.95th=[ 2311],
     | 99.99th=[ 2704]
   bw (  MiB/s): min= 9678, max=12220, per=100.00%, avg=10319.08, stdev=67.27, samples=956
   iops        : min=154848, max=195534, avg=165105.30, stdev=1076.36, samples=956
  lat (usec)   : 2=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%
  lat (usec)   : 750=0.01%, 1000=0.01%
  lat (msec)   : 2=99.58%, 4=0.41%, 10=0.01%, 20=0.01%
  cpu          : usr=3.81%, sys=96.17%, ctx=2138, majf=8, minf=2116
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=19807050,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=10.1GiB/s (10.8GB/s), 10.1GiB/s-10.1GiB/s (10.8GB/s-10.8GB/s), io=1209GiB (1298GB), run=120001-120001msec

Bunnies

I tried the same thing on the Bunnies zpool and got similar numbers meaning there’s a bottleneck somewhere or this is RAM speed. RAM should be way faster though?

These drives should be getting a max of 20GB/s reading from 34 of them. I can’t imagine why 34 drives would read at the exact same speed as 20 drives.

fio --filename=/mnt/Bunnies/Test/fio3.tmp --direct=1 --rw=read --bs=64k --ioengine=libaio --iodepth=64 --runtime=120 --numjobs=4 --time_based --group_reporting --name=throughput-test-job --eta-newline=1 --readonly
throughput-test-job: (g=0): rw=read, bs=(R) 64.0KiB-64.0KiB, (W) 64.0KiB-64.0KiB, (T) 64.0KiB-64.0KiB, ioengine=libaio, iodepth=64
...
fio-3.25
Starting 4 processes
Jobs: 4 (f=4): [R(4)][2.5%][r=10.2GiB/s][r=168k IOPS][eta 01m:57s]
Jobs: 4 (f=4): [R(4)][4.2%][r=9.93GiB/s][r=163k IOPS][eta 01m:55s] 
Jobs: 4 (f=4): [R(4)][5.8%][r=9.95GiB/s][r=163k IOPS][eta 01m:53s] 
Jobs: 4 (f=4): [R(4)][7.5%][r=10.0GiB/s][r=164k IOPS][eta 01m:51s] 
Jobs: 4 (f=4): [R(4)][9.2%][r=9.95GiB/s][r=163k IOPS][eta 01m:49s] 
Jobs: 4 (f=4): [R(4)][10.9%][r=9.99GiB/s][r=164k IOPS][eta 01m:46s] 
Jobs: 4 (f=4): [R(4)][12.5%][r=10.0GiB/s][r=164k IOPS][eta 01m:45s] 
Jobs: 4 (f=4): [R(4)][14.2%][r=9.88GiB/s][r=162k IOPS][eta 01m:43s] 
Jobs: 4 (f=4): [R(4)][15.8%][r=10.1GiB/s][r=165k IOPS][eta 01m:41s] 
Jobs: 4 (f=4): [R(4)][17.5%][r=10.0GiB/s][r=164k IOPS][eta 01m:39s] 
Jobs: 4 (f=4): [R(4)][19.2%][r=10.4GiB/s][r=170k IOPS][eta 01m:37s] 
Jobs: 4 (f=4): [R(4)][20.8%][r=11.2GiB/s][r=184k IOPS][eta 01m:35s] 
Jobs: 4 (f=4): [R(4)][22.5%][r=10.8GiB/s][r=177k IOPS][eta 01m:33s] 
Jobs: 4 (f=4): [R(4)][24.2%][r=10.7GiB/s][r=175k IOPS][eta 01m:31s] 
Jobs: 4 (f=4): [R(4)][26.1%][r=10.6GiB/s][r=174k IOPS][eta 01m:28s] 
Jobs: 4 (f=4): [R(4)][27.5%][r=10.0GiB/s][r=180k IOPS][eta 01m:27s] 
Jobs: 4 (f=4): [R(4)][29.2%][r=10.0GiB/s][r=180k IOPS][eta 01m:25s] 
Jobs: 4 (f=4): [R(4)][31.1%][r=10.5GiB/s][r=171k IOPS][eta 01m:22s] 
Jobs: 4 (f=4): [R(4)][32.8%][r=10.2GiB/s][r=167k IOPS][eta 01m:20s] 
Jobs: 4 (f=4): [R(4)][34.2%][r=10.1GiB/s][r=165k IOPS][eta 01m:19s] 
Jobs: 4 (f=4): [R(4)][35.8%][r=10.6GiB/s][r=174k IOPS][eta 01m:17s] 
Jobs: 4 (f=4): [R(4)][37.5%][r=10.4GiB/s][r=171k IOPS][eta 01m:15s] 
Jobs: 4 (f=4): [R(4)][39.2%][r=9.93GiB/s][r=163k IOPS][eta 01m:13s] 
Jobs: 4 (f=4): [R(4)][40.8%][r=10.1GiB/s][r=166k IOPS][eta 01m:11s] 
Jobs: 4 (f=4): [R(4)][42.5%][r=10.1GiB/s][r=165k IOPS][eta 01m:09s] 
Jobs: 4 (f=4): [R(4)][44.2%][r=9.00GiB/s][r=164k IOPS][eta 01m:07s] 
Jobs: 4 (f=4): [R(4)][45.8%][r=10.0GiB/s][r=165k IOPS][eta 01m:05s] 
Jobs: 4 (f=4): [R(4)][47.5%][r=9.84GiB/s][r=161k IOPS][eta 01m:03s] 
Jobs: 4 (f=4): [R(4)][49.2%][r=10.0GiB/s][r=164k IOPS][eta 01m:01s] 
Jobs: 4 (f=4): [R(4)][50.8%][r=9.00GiB/s][r=164k IOPS][eta 00m:59s] 
Jobs: 4 (f=4): [R(4)][52.5%][r=10.2GiB/s][r=167k IOPS][eta 00m:57s] 
Jobs: 4 (f=4): [R(4)][54.2%][r=10.3GiB/s][r=168k IOPS][eta 00m:55s] 
Jobs: 4 (f=4): [R(4)][55.8%][r=9.94GiB/s][r=163k IOPS][eta 00m:53s] 
Jobs: 4 (f=4): [R(4)][57.5%][r=10.1GiB/s][r=165k IOPS][eta 00m:51s] 
Jobs: 4 (f=4): [R(4)][59.2%][r=10.0GiB/s][r=165k IOPS][eta 00m:49s] 
Jobs: 4 (f=4): [R(4)][60.8%][r=9.93GiB/s][r=163k IOPS][eta 00m:47s] 
Jobs: 4 (f=4): [R(4)][62.5%][r=10.0GiB/s][r=164k IOPS][eta 00m:45s] 
Jobs: 4 (f=4): [R(4)][64.7%][r=10.2GiB/s][r=167k IOPS][eta 00m:42s] 
Jobs: 4 (f=4): [R(4)][65.8%][r=10.7GiB/s][r=175k IOPS][eta 00m:41s] 
Jobs: 4 (f=4): [R(4)][67.5%][r=10.3GiB/s][r=169k IOPS][eta 00m:39s] 
Jobs: 4 (f=4): [R(4)][69.2%][r=10.6GiB/s][r=173k IOPS][eta 00m:37s] 
Jobs: 4 (f=4): [R(4)][71.4%][r=10.8GiB/s][r=177k IOPS][eta 00m:34s] 
Jobs: 4 (f=4): [R(4)][72.5%][r=10.6GiB/s][r=174k IOPS][eta 00m:33s] 
Jobs: 4 (f=4): [R(4)][74.2%][r=10.5GiB/s][r=172k IOPS][eta 00m:31s] 
Jobs: 4 (f=4): [R(4)][75.8%][r=10.5GiB/s][r=173k IOPS][eta 00m:29s] 
Jobs: 4 (f=4): [R(4)][77.5%][r=9.98GiB/s][r=164k IOPS][eta 00m:27s] 
Jobs: 4 (f=4): [R(4)][79.2%][r=9.99GiB/s][r=164k IOPS][eta 00m:25s] 
Jobs: 4 (f=4): [R(4)][80.8%][r=9.89GiB/s][r=162k IOPS][eta 00m:23s] 
Jobs: 4 (f=4): [R(4)][82.5%][r=9.96GiB/s][r=163k IOPS][eta 00m:21s] 
Jobs: 4 (f=4): [R(4)][84.2%][r=9.84GiB/s][r=161k IOPS][eta 00m:19s] 
Jobs: 4 (f=4): [R(4)][85.8%][r=9.99GiB/s][r=164k IOPS][eta 00m:17s] 
Jobs: 4 (f=4): [R(4)][87.5%][r=10.0GiB/s][r=164k IOPS][eta 00m:15s] 
Jobs: 4 (f=4): [R(4)][89.9%][r=10.5GiB/s][r=172k IOPS][eta 00m:12s] 
Jobs: 4 (f=4): [R(4)][90.8%][r=10.8GiB/s][r=176k IOPS][eta 00m:11s] 
Jobs: 4 (f=4): [R(4)][93.3%][r=10.7GiB/s][r=176k IOPS][eta 00m:08s] 
Jobs: 4 (f=4): [R(4)][94.2%][r=10.0GiB/s][r=164k IOPS][eta 00m:07s] 
Jobs: 4 (f=4): [R(4)][95.8%][r=9.92GiB/s][r=163k IOPS][eta 00m:05s] 
Jobs: 4 (f=4): [R(4)][97.5%][r=9.93GiB/s][r=163k IOPS][eta 00m:03s] 
Jobs: 4 (f=4): [R(4)][99.2%][r=9.96GiB/s][r=163k IOPS][eta 00m:01s] 
Jobs: 4 (f=4): [R(4)][100.0%][r=10.2GiB/s][r=166k IOPS][eta 00m:00s]
throughput-test-job: (groupid=0, jobs=4): err= 0: pid=3058991: Wed Feb 15 14:13:06 2023
  read: IOPS=168k, BW=10.3GiB/s (11.0GB/s)(1232GiB/120001msec)
    slat (usec): min=4, max=8415, avg=22.86, stdev= 6.01
    clat (nsec): min=1520, max=11946k, avg=1499131.93, stdev=111897.50
     lat (usec): min=19, max=11971, avg=1522.09, stdev=113.33
    clat percentiles (usec):
     |  1.00th=[ 1205],  5.00th=[ 1303], 10.00th=[ 1369], 20.00th=[ 1434],
     | 30.00th=[ 1467], 40.00th=[ 1500], 50.00th=[ 1516], 60.00th=[ 1532],
     | 70.00th=[ 1549], 80.00th=[ 1565], 90.00th=[ 1598], 95.00th=[ 1631],
     | 99.00th=[ 1778], 99.50th=[ 1909], 99.90th=[ 2180], 99.95th=[ 2245],
     | 99.99th=[ 2540]
   bw (  MiB/s): min= 9810, max=13016, per=100.00%, avg=10511.75, stdev=116.03, samples=956
   iops        : min=156970, max=208258, avg=168188.04, stdev=1856.48, samples=956
  lat (usec)   : 2=0.01%, 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%
  lat (usec)   : 500=0.01%, 750=0.01%, 1000=0.01%
  lat (msec)   : 2=99.68%, 4=0.30%, 10=0.01%, 20=0.01%
  cpu          : usr=3.87%, sys=96.11%, ctx=2055, majf=0, minf=2106
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=20176896,0,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=10.3GiB/s (11.0GB/s), 10.3GiB/s-10.3GiB/s (11.0GB/s-11.0GB/s), io=1232GiB (1322GB), run=120001-120001msec
55

If you wrote fio3.tmp from dev/zero to your pool and used it for fio to read, then the result is what your pool is capable without cache in regards to throughput.

On my system that would definitely be the cache, but as I said, my system is already something for museum.

Your system should be able to saturate a 25Gb/s line at any time, without any tuning, special device or what ever.

Is it a shame not to use 100Gb/s Ethernet for such a rocket, Linux already installed? :slight_smile:

Edit: what I forgot to ask, did you cool the connect-x6 somehow, my card needs a fan

56

How many SAS3 cable (SFF-8643) has your backplane?

Your record size is default 128k, for dataset with movies or mp3 I would recommend at least 512k.

57

Didn’t you say they all come back to two SFF-8643 connections though? So that would be 24gb/s which is 3GB/s. Minus whatever overhead you have, SATA is something like 60MB/s in overhead because the bus is 600MB but real speeds are closer to 540MB

1 Like
58

There are 10 fans in my PC case, and 3 of them are coming up from the bottom, and 3 are pushing air out the top w/ one pulling out the back.

Airflow should be good. Is there a way to check?

I’d have gone 100Gb if I could. My switch maxes at 25Gb, and my NAS is max 2x10Gb at the moment.

I wouldn’t call this a rocket though. While the hardware should result in fast iops, it’s super slow relative to what you have.

59

My main NAS is direct-attach using a special-made board by the Storinator folks. That’s where “Wolves” is located.

The offsite one with HDDs uses a SAS Expander backplane with two SFF-8643 cables to each pool.

I added it in a previous message as a reference of another NAS with HDDs that I own in a similar configuration.

1 Like
60

I have 16 SAS cables plugged into my 3 LSI 9305 cards (6, 6, 4). Those are coming from the direct-attach boards on the Storinator XL60.

All drives are evenly spaced in both pools based on the ratio of drives per card among the different SAS ports, and each card maxes at 8GB/s. There should be plenty of bandwidth over 10GB/s to spare.

I’ve read a bunch about allocation size, and unless I’m using a database, the 128K size should be good enough for everything.

It doesn’t make sense to me, but that’s what I’ve read and seen in benchmarks.

Everyone who talks about ZFS optimizations is using VMs and Databases. I’m just storing files. I have Docker containers for VPN and Plex though.

Movies and Music are in different datasets, but everything is setup as 128K.

Thinking of this logically, 128K is 12.8K per drive in the pool which isn’t some clean number like 4K or 64K. Not sure if it matters though.