Return to Level1Techs.com

Home NAS ZFS Growing Pains

linux
server
zfs
#1

I’m attempting to upgrade my NAS hard drives from a 8x2TB raidz2 to a 4x6TB raidz1. The system is a Supermicro CSE-745 chassis with a Supermicro X8DTH-6 motherboard and a pair of Xeon x5687s with 48gb ECC RAM. I’m running Centos 7 with ZFS on Linux.

I’ve got the 4 new drives connected to a spare LSI 9211-8i powered off a spare EVGA PSU.

I keep having one or more of the new drives become “Degraded” with “too many errors”. Currently 3 of the drives are showing “too many errors” in when running zpool status and it’s already tried to resilver the pool. Also I’ve noticed that the “UDMA_CRC_Error_Count” on the 3 errored drives was increasing as it tried to resilver(went from 88 to 617 on /dev/sdb).

  pool: new6TBpool
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 75.9G in 0h25m with 0 errors on Tue Apr  2 01:56:21 2019
config:

        NAME        STATE     READ WRITE CKSUM
        new6TBpool  DEGRADED     0     0     0
          raidz1-0  DEGRADED     0     0 1.89K
            sdb     DEGRADED     0     0     0  too many errors  
            sdc     ONLINE       0     0     0  
            sdd     DEGRADED     0     0     0  too many errors  
            sde     DEGRADED     0     0     0  too many errors  

errors: No known data errors
Every 2.0s: smartctl -a /dev/sdb | grep UDMA_CRC_Error_Count && sm...  Tue Apr  2 01:57:43 2019

199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       617
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       74
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       659
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       650

I’ve tried using different Mini-SAS->SATA breakout cables, different ports on the card, and different PCIe slots. I’m beginning to think maybe the HBA is bad. I’m going to see if there’s a spare one at work I can borrow tomorrow to get this data transfer done.

Anything else I can try?

0 Likes

#2

First an observation on topology. Going from raidz2 2TB element to raidz1 6TB element seems retrograde. Less redundancy and higher probability of subsequent failure while rebuilding as you have a much larger element to resilver and the resilver take much longer thus longer exposure pool failure while awaiting a replacement disk since it is raidz1 and while resilvering since it is a bigger element. Less of an issue if you have a backup.

I agree another HBA would be interesting to check. If you could take your existing pool offline you could cross check with your know good HBA too, but I guess you don’t want to do that.

Could transplant the new pool to another server and generate test data to test load the pool?

I wonder about the second power supply. I haven’t ever done that myself, and don’t know the process, but would expect that a good common ground/0V connection would be required so the signalling between the PCI card and the SATA drives is good? I wonder if there is a difference in cabling between SATA vs eSATA or 8087 vs 8088 connectors which handles this potential issue?

You have ECC. I assume you have this is working OK and handling memory errors OK.

Have you checked ZOL github for issues on your version of ZFS?

Have you tried to setup a raidz2 pool rather than a raidz1 pool and see if that behaves differently?

Have you tried reducing the number of disks in the new pool to 3, 2, 1 and removing the unused data/power connectors and see if you have anything different?

0 Likes

#3

I’ve never had good luck with SATA drives in Supermicro chassis / drive cages. Sure they advertise SAS / SATA compatible, but out of four different builds, I always ended up buying hard drives twice (luckily someone else’s money 3x of the times). I would get the EXACT same problem you’re seeing - random errors on drives that test good.

0 Likes

#4

Yeah… I’d talked myself into going single parity to save a few hundred bucks, but the more I think about it I may just go buy another drive or two and run raidz2 again. Hmmm 6x6TB in raidz2 gets me an even 20TB usable… Plus I have a couple of spare bays if I want to do a replace one drive at a time upgrade in the future.

I don’t have access to another server with backplane etc… I could probably borrow a spare PC from work and that HBA I mentioned and build out the new pool on that. Heck then I could throw the 10g NIC out of my main desktop in it and transfer the data over that. Then I wouldn’t be dealing with the dual PSU thing either. I’ll work on doing this tomorrow evening, time permitting; work is crazy busy right now. :slight_smile:

Not sure… I do have both PSUs plugged into the same UPS, so their grounds should be at least close to each other.

Not having trouble with the Supermicro chassis/backplane. It’s been running flawlessly for a couple of years. It’s my ghetto pile of hard drives sitting outside the case on a spare PSU that’s giving me trouble. :joy:

0 Likes

#5

Rebuilt the the array on a different HBA in a spare PC from work. Probably going to get a couple more drives for the long term setup, but I’ll do some additional testing on this config for now. I popped my Mellanox 10g Infiniband/Ethernet card in it and started rsyncing from my main NAS. Getting 120-160MB/s on large files. Iperf was showing around 6 gigabit.

Going to let it run for a while and see if any errors develop; so far so good.

0 Likes