Wendell did some work on this here:
Linus had this weird problem where, when we built his array, the NVMe performance wasn’t that great. It was very slow – trash, basically.
This was a 24-drive NVMe array.
[image]
These error messages arent too serious, normally, but are a sign of a missed interrupt. There is some traffic I’m aware of on the LKML that there are (maybe) some latent bugs around the NVMe driver, so as a fallback it’ll poll the device if something takes unusually long. This many polling events, though, means the perf is terrible, because the system is waiting on the drive for a long, long time.
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1807393
I ran iotop, examined systat, used fio, bonnie, dd. I checked the elevator queue, the nvme queue, the # threads MD was allowed to use, and nothing seemed to work.
The system messages / kernel messages were filled with errors like
[80113.916089] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[80113.916572] vgs D 0 70528 2199 0x00000000
[80113.917051] Call Trace:
[80113.917531] __schedule+0x2bb/0x660
[80113.918000] ? __blk_mq_delay_run_hw_queue+0x14a/0x160
[80113.918466] schedule+0x33/0xa0
[80113.919078] io_schedule+0x16/0x40
[80113.919576] wait_on_page_bit+0x141/0x210
[80113.920054] ? file_fdatawait_range+0x30/0x30
[80113.920527] wait_on_page_writeback+0x43/0x90
[80113.920975] __filemap_fdatawait_range+0xaa/0x110
[80113.921416] ? __filemap_fdatawrite_range+0xd1/0x100
[80113.921851] filemap_write_and_wait_range+0x4b/0x90
[80113.922284] generic_file_read_iter+0x990/0xd60
[80113.922755] ? security_file_permission+0xb4/0x110
[80113.923225] blkdev_read_iter+0x35/0x40
[80113.923662] aio_read+0xd0/0x150
[80113.924090] ? __do_page_fault+0x250/0x4c0
[80113.924509] ? mem_cgroup_try_charge+0x71/0x190
[80113.924900] ? _cond_resched+0x19/0x30
[80113.925284] ? io_submit_one+0x88/0xb20
[80113.925665] io_submit_one+0x171/0xb20
[80113.926050] ? strncpy_from_user+0x57/0x1b0
[80113.926430…
Maybe their is something in their that can help.
You can get an inland professional SSD at the Micro Center for about $20-25 for your OS drive. They are good and reliable.
One thing to note is that ZFS does a lot of mem copy’s. Faster ram greatly improves performance. If you can’t get your home server running faster with other methods you may want to try a 3rd gen Ryzen and 3200 mhz ECC memory… Looks like you RAM is on the QVL. Double check your RAM settings are correct in the BIOS. That could make a big difference to and easily cause bad performance…