If you use Seagate Hard Drives in a Storage Pool, make sure you buy spares

…because Seagates Warrranty/RMA/Service department won’t take care of you anymore.

I’ve been buying Enterprise class hard drives from WD, Seagate and HGST for two decades now. Never before has the situation been this bad.

I’ve occasionally had drives start to fail. You contact the manufacturer, and because you bought enterprise products you had more options than with consumer stuff.

Most important of those was “Advanced Replacement” which is crucial for storage pools.

You give them the serial numbers of the failing drives, they overnight you replacements, you pop the replacements in the storage pool and resilver with zero downtime, and pop the failing drives back in the packaging they came in, and send them off to the manufacturer. Simple.

Then maybe a decade ago they started charging for advanced replacement. it wasn’t much. Probably just enough to cover the extra shipping. It annoyed the hell out of me that I now had to pay for something that previously was part of the deal when you bought enterprise gear, but I grumbled and accepted it, because I didn’t have much of a choice.

Now it is utter nonsense. The online automated RMA form only offers “standard replacement” with a 10-14 day replacement timeline. (You have to send them the drives before they even start processing the claims)

This just won’t work. The pool can’t sit in a degraded state for that long. That is super risky.

So, I figured I’d just call them. I’ve spoken to Seagate customer service many times over the years. Maybe they are just hiding the advanced replacement service behind a phone call now?

…but you call the 800 number and there is no way to get a person on the line for drives. They inform you that the only way to get service is via a chat on their website.

So I suck it up and go on the chat.

…and I can’t even tell if I am speaking to actual human beings. Chances are I am just being bounced between chatbots. And the outcome is unchanged. Warranty is now only offered via standard replacement.

I don’t understand how one can operate a hard drive business like that. Enterprise drives are always going to be used in storage pools. No operator of a high criticality ZFS pool is ever going to be able to tolerate having a degraded pool for 10-14 days while they wait for a replacement.

Essentially they have rendered their warranty completely worthless, forcing customers to buy new drives to use while they RMA the failing ones. It feels like a bait and switch. Advanced Replacement and other premium service options was one of the reasons you bought enterprise drives. I’ve spent thousands on enterprise drives, and now all of a sudden, none of the service options this used to get you are available anymore.

It’s enough to make you want to burn the whole damn company to the ground.

9 Likes

I mean, this is kind of the new reality now. When you can count the number of “real” customers that they have on one hand, customer service suffers, the sad reality is you aren’t their customer anymore. They don’t care because they don’t need your business.

As soon as I notice a failure in any of my pools, I go out and buy a spare. I’ve always done that out of personal convenience and having an extra hard drive after they return it under warranty is not a bad thing. I always find a use for those. But lately the stock is so light I can’t even do that. So I have a brand new unused drive sitting on my shelf right now.

For the record, I don’t like this either.

2 Likes

Many such cases. I would not just single out any single vendor. Everybody will do their best to turn you down nowadays.

1 Like

Here in the EU, i have witnessed multiple cases where there was no warranty on drives because they were “OEM”.

Or meant for a different region.

Or weren’t bought at the right shop.

Buying new is getting harder as well.

2 Likes

An enterprise should always have a stock of spare drives. Their arrays/pools should be built with hot spare capacity. That’s why a lot of local hardware storage arrays use a RAID6 or similar proprietary system (like Dell ADAPT) by preference. It’s also why enterprises like SANs, but that’s a different conversation for a different day. :wink:

Smaller organisations can often rely on just the hot spare capacity, but that’s not preferable given the increase in shipping times these days. That lesson is often a hard one to learn.

Sometimes I think that the phrase “Enterprise” is actually just marketing aimed at small businesses with pretentions. This may be one of those cases. :wink:

Whilst this fits the pattern of enshittification, I suspect it’s something that they can do because their actual enterprise and SME customers weren’t relying upon it.

Sure, they may have been using it, but they didn’t need it. If a drive fails they replace it from a pile of spares sitting on a shelf. By the time that a person is physically there to replace the failed drive a hot spare will probably have been completely populated anyway. In a large enough infrastructure you might even schedule this as a weekly sweep to collect & replace failed disks and other hardware.

A fourteen day shipping period doesn’t matter in that situation, so long as the reserve stock of drives on a shelf is able to last fourteen days at twice the normal failure rate in your infrastructure.

So it’s frustrating, but it’s also somewhat inevitable. This service was a hangover from the 90s when enterprises were still figuring lots of things out. Now they have this figured out, and it turns out that they don’t need this service.

If spare drives aren’t part of your storage strategy (and at current costs that’s understandable) then you need to at least add more copies of the data to your strategy to compensate. That’s difficult for volatile data, but a lot of what we store is fairly static and for that data it’s fortunately much easier.

ETA - the customer service & chat / chatbot issue is entirely enshittification, and bloody annoying. Absolute crap, up with which we should not put. Whilst I’ll defend them dropping the overnight rush replacement service, I won’t defend modern customer service trends. They can take their chat features and AI bots and RMA the damned things. And may the devil himself take their support tickets and place them in the lowest priority queue! :face_with_symbols_on_mouth:

2 Likes

Ill add that the support contracts from the usual suspects (dell hpe lenovo etc) usually cover failed drives. If you are using their drives anyway
A lot of smaller businnesses dont have spares but the server vendor does (or at least they did before the current shortage), and they could send a tech to swap the drive or just ship the drive in a day or so.

1 Like

Whilst i feel your pain, no sensible enterprise array administrator is going to configure zero hot-spares? Typically my aggregates are RAID6 (or RAID-DP, RAIDZ2 or equivalent) with 1 hot spare per 16-24 drives. i.e., if they’re spaced out enough for one re-sync, i can tolerate 3 drive failures. On my remote standalone machines, they’re a mirror with 1 hot spare configured.

If one drive failure puts you in a degraded state until you can source a replacement drive (and this is an “enterprise” setup - for home, run raid0 or whatever you like, just have backups), your pool configuration is wrong. Configuring one or more hot spares per aggregate has been standard for at least a decade or two at this point?

edit:
IIRC my ancient Dell/Equalogic PS5000 array from… 2008(?) shipped from Dell with a 14+2 configuration - 14 drives in RAID10 + 2 hot spares.

2 Likes

Hot spares are a pretty poor approach, as your spare drives wind up wearing motors and bearings just like your active drives.

Though to your point, having cold spares on the shelf is probably something a business should do. At least now, when you can’t get an “Advance RMA” overnighted to you.

Pretty sure they don’t spin up until needed?
I mean maybe every enterprise array vendor is wrong tho…

edit:
looks like they do spin - and apparently NOT spinning for extended period can cause bearing wear so… basically replace drives when they get old :smiley:

My pool is ZFS, RAID60 equivalent. Two VDEV’s each with two redundant drives.

The problem is that despite my attempts when buying my last set of drives to spread them out between production lots and shipments, three drives out of nowhere started developing issues at the same time. One outright dropped out while I was in the process of addressing it, but the others appear to be holding on for now (while I am attempting to address them too)

I do have backups (at least of the impossible to replace stuff, the media library is a different story) but having a pool go down is still a major nuisance which is best avoided.

I guess it works for them because the cost of support so greatly outweighs the cost of drives, that they’d rather throw more drives at it, and if they fail they fail, at least they won’t need to pay for support.

They live in a completely different world than the small to medium business and home lab users.

1 Like

Well that sucks - how old are the drives? environmental conditions OK?

I had a server room in the Pilbara hit 54 degrees Celsius (! 6-10 degrees hotter than outside :D) when the air conditioning failed back in the mid 2000s, mysteriously had a few drive failures the next week that EMC replaced :smiley:

irony: first device to fail in the server room with the e-bay temperature monitoring probe. first knowledge of the failure was “wtf is the server room so loud for?” :smiley:

On traditional RAID, yes.

On more modern systems like Dell’s ADAPT (which I alluded to earlier) they spread the spare space over the drives just like they’re spreading the data and the parity information. If a drive dies it can be repopulated from the parity into the spare space on the other drives, and no drive is ever inactive or completely unused once in a volume/pool/array/$nomenclature

I believe most SAN systems do similar things. All very clever. All above my pay grade, if I’m honest. I’m a humble sysadmin, not a storage expert, but the storage guys I’ve worked with explained it in suitably short words for me. :wink:

It’s the best of both worlds - lowers the risk of bearing seizure/failure due to inactivity, and also gets usage out of each drive.

Currently having a bad experience with WD SSD warranty. I bought them before WD SSD’s were merged with Sandisk and spent 3 days arguing with sandisk tech support that they are legitimate drives. Shipped the drive out and had to contest the delivery of the item even though their own shipping label showed it had arrived 3 weeks ago. They are still dragging their feet replacing the drive. I’m never buying from them again if possible. :face_with_symbols_on_mouth:

Will say had a good experience with a mushkin RMA from earlier this year. Whole process took about 2 weeks with shipping. So guess I’ll be buying from them for the foreseeable future.

About 4 years old. ~36k hours logged in SMART. Still within 5 year warranty.

They have been running 24/7 with only minimal shutdown and startup cycles.

They are in a Supermicro 4U chassis (24 drive model) but only half populated (12 drives) staggered so no two drives are directly over, below or next to eachother. Unused bays have caddies with blocked vents so all airflow is forced over the drives.

190 Airflow_Temperature_Cel 0x0022 068 056 000 Old_age Always - 32 (Min/Max 26/38)
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 32

Hottest they have ever been is 38C, but that was an extreme outlier. Most of the time they sit at ~32C to 33C.

The use environment does not have any severe shock or vibe conditions. It’s a pretty standard (albeit slightly older) rackmount install.

Full data of the one that really failed:

ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   083   064   044    Pre-fail  Always       -       0/185295720
  3 Spin_Up_Time            0x0003   091   091   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       37
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       6
  7 Seek_Error_Rate         0x000f   088   060   045    Pre-fail  Always       -       0/607949711
  9 Power_On_Hours          0x0032   059   059   000    Old_age   Always       -       36611
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       37
 18 Head_Health             0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0 0 0
190 Airflow_Temperature_Cel 0x0022   068   056   000    Old_age   Always       -       32 (Min/Max 26/38)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       32
193 Load_Cycle_Count        0x0032   098   098   000    Old_age   Always       -       5670
194 Temperature_Celsius     0x0022   032   044   000    Old_age   Always       -       32 (0 21 0 0 0)
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       21
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       21
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Pressure_Limit          0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   100   000    Old_age   Offline      -       36004h+50m+33.986s
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       152061023539
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       71932562332520

Environment wise, nothing seems to be truly out of the ordinary. Pretty ideal conditions from my experience.

I detected the problem starting when I saw 2 reallocated sectors and 30 pending sectors, and then by the time my first replacement drive arrived it had been dropped from the pool by ZFS due to repeated read errors.

It looks like classic media degradation.

Two other drives have single digit reallocated sectors, and I am going to try to get them replaced before they start degrading like this first one did.

First replacement drive is undergoing initial testing right now, and will be resilvered in in the next two days when it is done.

I’m just doing a single fio pass on this one. I usually do badblocks, with four read/write passes, but badblocks can’t handle more than 4.3B (32bit) blocks, so it is no longer useful on larger drives.

I was going to do four fio passes, but that is when I noticed that the bad drive had dropped out of the pool, so I decided to shorten it to a single pass to get more redundancy back in the pool before anything else fails.

2 Likes

thanks for letting us know

He didn’t single them out. He’s a long time customer of Seagate and they’re screwing him. He’s not being unfair.

That’s a cold spare. Cold spares are riskier because there’s a chance the drive will fail to spinup and/or initialize properly when needed.

Besides building drives with more capacity who is actually building a new device that can hold a ton of data on a storage medium that’ll last 100 years plus?

There isnt much. Most of that died with optical drive technology.

Microslop is running Project Silica, where they store data by 3D etching glass, and thats effectively eternal (the machine to read it is not). But who knows if and when that will turn into useful technology in the hands of the people