Extremely slow copy between two fast NVMe's

[twin_savage]

I ended up roughly quantifying the behavior in another thread; thought it might be useful here so copying the post over:

@twin_savage

It was bothering me that I didn’t quantify how bad NAND gets over time so I did a proper test of degradation:

Test was performed on a PE image that wasn’t using firecuda 520 for any other ancillary OS reads/writes.

Read test performed on SSD with “aged” data:

A “surface test” was performed on SSD to refresh all NAND cells

Read test performed again with refreshed data on SSD:

​​​ ​ ​

​​​ ​ ​
As you can see in the bottom righthand corner the average read speed for the entire disk went from 193 Mb/s to 2519MB/s. For the past 2 years GC had been running in the back ground on SSD and trim was run regularly so they do not factor in.

An interesting point is that I cleared off ~500GB of files before running the test in the hopes that it would run faster compared to the test in the previous post and the average read speed did improve, but perhaps that is just because the majority of the 500GB were old files.

This behavior is typical of every SSD I have ever used (many dozens of different SKUs if not hundreds) except for some of the early Ultra-320 SCSI SSDs.