Home NAS ZFS Growing Pains

I’m attempting to upgrade my NAS hard drives from a 8x2TB raidz2 to a 4x6TB raidz1. The system is a Supermicro CSE-745 chassis with a Supermicro X8DTH-6 motherboard and a pair of Xeon x5687s with 48gb ECC RAM. I’m running Centos 7 with ZFS on Linux.

I’ve got the 4 new drives connected to a spare LSI 9211-8i powered off a spare EVGA PSU.

I keep having one or more of the new drives become “Degraded” with “too many errors”. Currently 3 of the drives are showing “too many errors” in when running zpool status and it’s already tried to resilver the pool. Also I’ve noticed that the “UDMA_CRC_Error_Count” on the 3 errored drives was increasing as it tried to resilver(went from 88 to 617 on /dev/sdb).

  pool: new6TBpool
 state: DEGRADED
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: resilvered 75.9G in 0h25m with 0 errors on Tue Apr  2 01:56:21 2019
config:

        NAME        STATE     READ WRITE CKSUM
        new6TBpool  DEGRADED     0     0     0
          raidz1-0  DEGRADED     0     0 1.89K
            sdb     DEGRADED     0     0     0  too many errors  
            sdc     ONLINE       0     0     0  
            sdd     DEGRADED     0     0     0  too many errors  
            sde     DEGRADED     0     0     0  too many errors  

errors: No known data errors
Every 2.0s: smartctl -a /dev/sdb | grep UDMA_CRC_Error_Count && sm...  Tue Apr  2 01:57:43 2019

199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       617
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       74
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       659
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       650

I’ve tried using different Mini-SAS->SATA breakout cables, different ports on the card, and different PCIe slots. I’m beginning to think maybe the HBA is bad. I’m going to see if there’s a spare one at work I can borrow tomorrow to get this data transfer done.

Anything else I can try?

First an observation on topology. Going from raidz2 2TB element to raidz1 6TB element seems retrograde. Less redundancy and higher probability of subsequent failure while rebuilding as you have a much larger element to resilver and the resilver take much longer thus longer exposure pool failure while awaiting a replacement disk since it is raidz1 and while resilvering since it is a bigger element. Less of an issue if you have a backup.

I agree another HBA would be interesting to check. If you could take your existing pool offline you could cross check with your know good HBA too, but I guess you don’t want to do that.

Could transplant the new pool to another server and generate test data to test load the pool?

I wonder about the second power supply. I haven’t ever done that myself, and don’t know the process, but would expect that a good common ground/0V connection would be required so the signalling between the PCI card and the SATA drives is good? I wonder if there is a difference in cabling between SATA vs eSATA or 8087 vs 8088 connectors which handles this potential issue?

You have ECC. I assume you have this is working OK and handling memory errors OK.

Have you checked ZOL github for issues on your version of ZFS?

Have you tried to setup a raidz2 pool rather than a raidz1 pool and see if that behaves differently?

Have you tried reducing the number of disks in the new pool to 3, 2, 1 and removing the unused data/power connectors and see if you have anything different?

I’ve never had good luck with SATA drives in Supermicro chassis / drive cages. Sure they advertise SAS / SATA compatible, but out of four different builds, I always ended up buying hard drives twice (luckily someone else’s money 3x of the times). I would get the EXACT same problem you’re seeing - random errors on drives that test good.

Yeah… I’d talked myself into going single parity to save a few hundred bucks, but the more I think about it I may just go buy another drive or two and run raidz2 again. Hmmm 6x6TB in raidz2 gets me an even 20TB usable… Plus I have a couple of spare bays if I want to do a replace one drive at a time upgrade in the future.

I don’t have access to another server with backplane etc… I could probably borrow a spare PC from work and that HBA I mentioned and build out the new pool on that. Heck then I could throw the 10g NIC out of my main desktop in it and transfer the data over that. Then I wouldn’t be dealing with the dual PSU thing either. I’ll work on doing this tomorrow evening, time permitting; work is crazy busy right now. :slight_smile:

Not sure… I do have both PSUs plugged into the same UPS, so their grounds should be at least close to each other.

Not having trouble with the Supermicro chassis/backplane. It’s been running flawlessly for a couple of years. It’s my ghetto pile of hard drives sitting outside the case on a spare PSU that’s giving me trouble. :joy:

Rebuilt the the array on a different HBA in a spare PC from work. Probably going to get a couple more drives for the long term setup, but I’ll do some additional testing on this config for now. I popped my Mellanox 10g Infiniband/Ethernet card in it and started rsyncing from my main NAS. Getting 120-160MB/s on large files. Iperf was showing around 6 gigabit.

Going to let it run for a while and see if any errors develop; so far so good.