I have 12 Seagate Exos X20 20TB SAS drives and 3 have developed an odd problem. (Two of the three bad drives are replacements from Seagate.)
The drives normally have a transfer rate from about 260MB/s at the beginning of the drive to about 120MB/s at the end. The three bad drives have a constant transfer rate of 86MB/s across the entire drive. They used to perform normal but now they are slow.
I tested and completely reformatted them using the SeaChest tools but that made no difference in performance. There are no errors being reported, just slow.
Has anyone else run into this? Are there any remedies I can try?
They should all be CMR drives. The part numbers of all 12 drives are identical. The three “bad” drives used to run at full speed and then just slowed down. It’s funny that all three problem drives run at exactly 86.7MB/s. It’s like the interface or something is rate limited.
These drives have been tested outside of my array. I have tried different controllers, different cables and even different Linux kernels. (I don’t have anything with Windows on it right now.)
There should be plenty of airflow. SeaChest Diagnostics reports no drive has ever been above 37*C.
Since you’re talking seachest, I assume they’re already on the latest firmware.
Is it the same speed across the entire platter? How do you measure the speed?
How do IOPS look compared to the healthy drives?
If possible, try changing the sector size and see if there is any observed difference in performance, even if minor?
I do believe that all the drives are running the same and latest firmware. (E006)
I never thought to do an IOPS test. I was just doing sequential write tests using Gnome Disk’s built in benchmark and the “dd” and “time” commands.
I had originally formatted these disks for 4K sectors. I am currently reformatting to 512 sectors to see if that makes a difference. Looks like this is going to take about 3 days to complete. The other “working” drives complete the same task in 24 hours.
I was doing all of my testing on the root device itself and not through any filesystem. Gnome Disk Benchmark seems to read chunks evenly across the entirety of the drive which usually gives me a reasonable map of drives performance across the surface. I used “dd” with the seek parameter to force it to write at different spots. (Maybe this doesn’t work like I think.)
When the drives are done re-formatting I will try benchmarking again and see what happens.
This might be an added question that just shows my inattention (such as not reading your posts carefully enough - sorry), but: are all those drives in the same enclosure and if so, is that enclosure something like a DAS? I have such a setup, which is connected to my PC by USB 3, and noticed a similar behavior with some, but not all drives. In my case, the slower drives really slow down a lot if I copy between drives in the same enclosure. I am (for now) blaming mainly the Asmedia controller (with updated firmware, didn’t change anything) of the DAS. Curiously enough, the “slow” drives are plenty fast if I connect them to an individual USB-to-SATA adapter, and transfer files through that. In contrast, switching the drives around inside my enclosure didn’t change anything (?). It also has two fans, no dust and plenty of airflow.
Even if your setup is quite different (e.g. NAS or similar), it might be worth your while to take one of your slow drives, connect it to a USB-to-SATA adapter (external enclosure) and run a quick speed test. If nothing else, if the drive is still slow, it eliminates the SATA controller/backplane from the list of suspects.
And yes, especially with the current price spikes for drives, drives behaving strangely is really annoying.
Ultimately they will be part of a ZFS RAID-Z2 array. But at this moment, I am testing the drives without any filesystem on them.
My setup is a homemade NAS system. The drives are connected to an LSI 9305-16i SAS HBA. (The drives are the SAS models.)
I have tried these drives in a different computer running a different SAS HBA but get the exact same results. Thus, I don’t believe it is a problem with the hardware other than the drives.
I don’t have a USB-to-SAS adapter that works to try. (I have bought a few and they all seem to suck.)
Completely annoyed with the prices right now. I was just going to buy another drive, but quickly did an about-face since they are now double what I paid last year for my current drives.
That’s weird… I think my sector switch tests took minutes (on 18TB exos), not hours. It should not be involving any data operations… unless the maybe the drives are not zeroed and the firmware is trying to preserve existing data.
You’re using Gnome disks? That’s a GUI utility, so maybe post a few screenshots? Healthy vs slow to compare…
What about SMART tests? Do the short and long ones complete with no issues?
The pattern of slow-down is also weird - going from decreasing speed to constant across the platter. A slower motor or something of the sort would preserve the slope of the graph, although I’ve never heard of HDD motors randomly slowing down…
This is such an irregular failure mode, it’s hard to even spitball potential causes.
.