I have 4x Lexar SSD NM790 4TB in a RAIDZ1 ZFS pool. I have been doing daily scrubs for last 3 years. Last week I noticed that life used was at 3% while I had only written like 20TB over the last 3 years. That got me digging deeper into logs/counters exported by the SSD.
What I found was that the media units written was substantially (I mean that) larger than host writes.
$ sudo nvme endurance-log /dev/nvme0 -g 0 | grep written
data_units_written : 18809
media_units_written : 348243
Now, that is 18x larger!
So, I did some experiments:
(1) Noted how MUW changes from before the daily scrub, during scrub and after scrub, with no other traffic to pool or the drives.
(2) Did ‘dd’ read of one of the drives for 640GB (go past the SLC cache size) with no other operations happening on the drives.
Both of these resulted in very large media writes to the NAND and went on for hours (it will write like 20GB in 5 mins and go on for 3-4 hours). A scrub read of 2TB used on the drive would result in anywhere around 400-800GB of NAND writes. A 640GB ‘dd’ read of the SSD lead to 340GB of NAND writes.
You can notice this right away: as soon as you have read “enough”, the read slows down to half the speed because I notice that MUW starts to go up i.e. the SSD controller is doing some sort of maintenance writes to the NAND and moving stuff around.
Now, this came as a surprise to me because I always thought reads were free, the scrub was free (and fast, hence daily for this NVME pool). But I have been wearing out my drives by doing that.
I have heard of read-reclaim relocation where the SSD controller does move blocks around when a read threshold is reached or it encounters some ECC errors and thinks the block needs refresh. But this is bonkers kind of NAND writes for a large sequential read.
Another angle: The NM790 uses a very large dynamic pSLC cache. At ~50% mapped occupancy, theoretical pSLC capacity is ~640 GB. Could MAP1602 be staging read-reclaim relocation into pSLC and subsequently folding it to TLC, causing roughly 2× physical programming for some reclaimed data? Is anyone familiar with MAP1602 behavior or able to identify some of the vendor telemetry counters exposed by log IDs 0x07 and 0x08?
Has anybody noticed something similar on their SSD system? Can someone please run some tests?