Logs checked, nothing there.
Either way I’ll give it another go in the morning, it’s almost 1am and I need rest, this can wait. I’ll try a Linux copy tomorrow, maybe that works.
Thank you all for the feedback, if you have any other suggestions, please do let me know and I’ll pick it up in the morning
maybe there is some useful information
ADHD and curiosity got the better of me so I booted up a quick PopOS live disk, it’s almost the same behaviour, except here it stays around 15-20MB/s instead of 3-5MB/s, but exact same thing. Randomly pops up in speed as well but it seems to have stabilised around 30-ish MB here after about 10 minutes of copying. I think we can agree that’s well below the 3GB read speed of the source and 5-6GB write speed of the target.
Newest windows patch borked SSD speeds for some people. Hopefully they fix it soon.
Not sure if this has been suggested, have you checked the drive temps? If it’s directly under the GPU, or on the back of the motherboard, it’s very possible that it’s thermal throttling. I’ve seen NVME drives boil themselves at over 100c, but most of the controllers seem to be rated for, and throttle to stay at/near 70~80c or so.
I hope this could help:
I’ve left it to copy overnight and when I came back this morning, I think the PC froze or something because it wouldn’t wake the screens up, this was in Linux. Either way it seems to have copied most of the files, I’m just letting FreeFileSync compare the two drives and see if/what is missing.
EDIT: It still hadn’t copied all files, must have only been around 2/3 of the way through, still 250GB to copy out of 750 
Temps are fine, never go above 55-60C even under stress, they’re well cooled.
As for the updates, I did look into it but again, this insane slow-down does not manifest in crystalDisk nor Magician, and only on this one drive. I was thinking maybe I don’t have enough free space, these drives slow down to a halt when full but I still have more than 10% available, if not almost 20.
I am surprised noone has mention DRAM-cache yet. What I suspect is happening is that the large copy runs out of DRAM cache and that slows things down to a crawl. Of course, could be entirely wrong here, but…
If that was the case, surely it wouldn’t be a problem with copying 100-200MB from one drive to another? I will look into the update as well but in my mind, logic dictates it’s a device-specific problem, aggravated by the OS.
Is there a way to flush the drive caches and such?
From my knowledge caching problems come from one of two ways:
With 200 MB sloooooooowness it sounds like #2 is in effect, but it could also be something completely else. My gut instinct is caching problems, but the only thing I do know is that it should not behave that way.
That’s informative, thank you, my suspicion is the same, but that’s why I always allow over-provisioning, drive is only 3/4 full, I thought 10% was sufficient OP. The part that really bums me is why are there no SMART warnings, no errors, and synthetic tests like Crystal or Magician always come up with great speeds?
That said, what I’ll do after this copy is just empty the drive, delete the partition entirely, make a new one, trim the drive just in case, if it can even do that, and see if the problem persists.
One thing I’ve noticed is that the speeds jump significantly at the beginning of each file, doesn’t matter if it’s 10MB, 1GB or 50, it’s the same, starts off at 300+, dies down immediately after. It’s still not good though. This should be at minimum 10x faster, if not 100 on certain files.
For example large photogrammetry scan folders, 100’s of RAW photos off a camera - absolute destroyer, 4-5MB at best.
If the drive starts to behave normally after the re-partitioning and writing data back to it, that would strongly suggest the “old” data scenario (NAND charge leakage).
I have a bit of an update, your Corsair Toolbox idea half paid-off. TRIM is disabled because it’s handled through windows, so I can’t do it from the app. However, I found I was on the second-to-last firmware, so I updated it. While still slow, comparing content between source and target now happens around 25-35MB/s, still dips to around 10 occasionally but not for long, far cry from the 2.5MB/s woes from just before. I imagine this would go even faster in Linux. It’s still junk speed but considerably faster still than before. I see now every time it wants to go under 10MB it probably flushes cache and goes right back up.
This is sounding very reminiscent of the whole 840 evo debacle. I wonder if these same mechanism could be behind it.
I suspect you were right on point with that charge drift theory. After a volume delete/re-make and allowing 10% over-provision and re-format, it’s back to the advertised speeds.
I’ve copied some files back to the drive, with a few exception hick-ups where it immediately dipped down to a stand-still, it wrote consistently with around 1.2GB/s once it got past those, even going as high as 2GB/s, copied an 85GB project folder in a minute or so.
So, now that I know this is a possibility, is there anything to be done to avoid this problem?
I appreciate all your inputs, this is actually educational in NVMe/SSD behaviour.
Regarding avoiding the bad behavior in the future, there are utilities that will test drives by copying data off of a block, writing a test pattern and reading it back to verify everything is in order, then copying the original contents back to the block.
These utilities can be used to “refresh” the data on NAND, but be aware you’re going to burn P/E cycles on the SSD by using them. I think if used sparingly they would be okay to use.
badblocks is an example but its linux only. Be extra careful when getting one of these tools for windows because there seems to be a large presence of malware masquerading as these utilities out in the wild.
I think this behavior is news to the majority of people on this forum with how often I hear people promoting SSDs for long term data storage.
It is theoretically possible to construct an SSD controller that would do the refreshing of weak cells in the background, but I don’t know of any SSDs that do this, instead the mechanism most SSDs use to try to handle charge decay is by making cell voltage offset tables to map what it reads as weak voltages to a proper voltage levels, this works until the charge levels start to overlap with the resolution of the cell voltage sampling circuit. Lots of ECC schemes are used in conjunction with these voltage offset tables so that data isn’t as easily lost.
There is actually a new mechanism of less predictable cell voltage decay in 3D NAND that throws a wrench in dealing with cell charge decay in this manor though; we’ll start seeing soon if 3D NAND has more problems or not.
I don’t work in SSD design so it is entirely possible that some of the manufactures have come up with some clever way of dealing with the problem but keep it a trade secret.
Thanks for sharing this
I have an MP600 that is behaving strangely similar.
I will try this before RMA!
Thanks
Henrik
FWIW I have a firecuda 520 that displays the same behavior.
Ohh 
I am not happy to hear that I have two of them too! 1Tb.
They have for me been very reliable as my boot drive
I ended up roughly quantifying the behavior in another thread; thought it might be useful here so copying the post over: