We’ve encountered an interesting issue with a specific NVMe model that doesn’t occur with any of our other NVMes. I’m wondering if this is a known issue or if there’s a plausible explanation.
The sequential read bandwidth drops from ~5 GiB/s to ~3 GiB/s when the read concurrency is high. We’ve tested this with fio, and the issue arises when we increase the number of concurrent threads or iodepth. Once the cumulative count exceeds 256, the disk starts lagging.
(the 2nd picture is capped and the axis a bit wrong, the 3rd is uncapped)
This issue consistently occurs on our Samsung MZ1L23T8HBLA-00A07 drives (of which we have many), but not on the MZQL23T8HCLS-00A07, which is the same PM9A3 series but in U.2 form factor.
My only hypothesis is that U.2 supports two power states (25W and 8W), while M.2 only supports one (8W). I switched the U.2 drive to 8W, and its sequential read dropped to ~3 GiB/s, though the read profile differs from the M.2. In the U.2 drive, the speed remains consistent regardless of the iodepth or thread count.
I haven’t observed this kind of performance inconsistency with other SSD/NVMe drives. So, what could be causing the sudden speed drop on the M.2 version? Is it possibly related to power consumption and throttling? Maybe it’s something simple that I’m overlooking. Does anyone have any ideas?
M.2 is really not designed for the enterprise space for high q depth applications, as the form factor is not readily hot swappable by most deployments. As such, concessions are made and hopefully unnoticed (you did)
This is controller limitations to mitigate heat and meet form factor criteria.
U.2/U.3 and E.1/E.3 are readily hot swappable and have larger heat capacities by design so they should not have the same design constraints imposed.
I’ve conducted additional tests on our older M.2 drives, Samsung PM983, PM963, and (even) Micron 5100. The performance results were as expected- steady growth without any sudden drops. Interestingly, the PM983 showed improved performance with an increased thread count, in contrast to the PM9A3. (Perhaps we’ve installed them upside down? :^) )
(tried to attach Micron graph, but the limit is 2 pics. nothing interesting there anyways)
It seems that queue depth is not a significant issue for M.2 drives in general, and our finding in 9A3 is an exception. Maybe I’ll be able to tune the queuing in the OS, seems like the speed degrades when the queue (?) is nearly full.
As for thermal throttling, we haven’t seen any issues with the thresholds being surpassed. I’m planning to install a heatsink, so we’ll see if that makes a difference.
