Need help with mdadm raid [raid5]

Software & Operating Systems Linux
,
1

I have a raspberry pi setup as a NAS. Hooked up a 5 Bay Orico Sata to USB enclosure with 4x4 TB Seagate IronWolf drives in it. Everything was working till some time ago but post shifting to my house when I booted everything the raid was not accessible. I was showing 1 drive missing. Then I opened up the enclosure cleaned it up and put everything back in. I then ran the force assemble command and I was able to get the drive working.

After that I started playing around with removing the drive by tagging it as failed drive to try and get to re-adding it. Seems like I messed up pretty bad. I am now in a position where the raid seems to be back to how it was but it is not mounting at all. When I try to mount it says can’t read superblock on /dev/md0. I have half a mind to say f* it and just give up and start from scratch but I wanna try and give it one last go in hopes that I can recover the data.

Outputs of a few commands to show where the raid is at:

Drive info:

pi@pi:~ $ blkid
/dev/sdb: UUID="6ee9d50a-6f21-5cef-9e5d-e359d806fd4c" UUID_SUB="a76d62e3-9bf9-c38c-b4dc-ee8fb58cd1e8" LABEL="pi:0" TYPE="linux_raid_member"
/dev/sda: UUID="6ee9d50a-6f21-5cef-9e5d-e359d806fd4c" UUID_SUB="b835def0-00fd-62dd-13e2-a26520a3c513" LABEL="pi:0" TYPE="linux_raid_member"
/dev/sdd: UUID="6ee9d50a-6f21-5cef-9e5d-e359d806fd4c" UUID_SUB="d2048f05-ac91-96fe-30fc-31ae802be8a7" LABEL="pi:0" TYPE="linux_raid_member"
/dev/sdc: UUID="6ee9d50a-6f21-5cef-9e5d-e359d806fd4c" UUID_SUB="4a1653ce-1b8b-eeda-c91d-6a4b576df662" LABEL="pi:0" TYPE="linux_raid_member"
/dev/sde1: LABEL_FATBOOT="boot" LABEL="boot" UUID="54E3-79CE" TYPE="vfat" PARTUUID="637e2230-01"
/dev/sde2: LABEL="rootfs" UUID="c6dd3b94-a789-4d57-9080-1472f721804b" TYPE="ext4" PARTUUID="637e2230-02"
pi@pi:~ $ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda      8:0    0   3.7T  0 disk
sdb      8:16   0   3.7T  0 disk
└─md0    9:0    0  10.9T  0 raid5
sdc      8:32   0   3.7T  0 disk
└─md0    9:0    0  10.9T  0 raid5
sdd      8:48   0   3.7T  0 disk
└─md0    9:0    0  10.9T  0 raid5
sde      8:64   0 167.7G  0 disk
├─sde1   8:65   0   256M  0 part  /boot
└─sde2   8:66   0 167.4G  0 part  /

Raid examine info:

sudo mdadm --examine /dev/sd[a-e]
/dev/sda:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6ee9d50a:6f215cef:9e5de359:d806fd4c
           Name : pi:0  (local to host pi)
  Creation Time : Mon Nov  2 23:52:14 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7813772976 (3725.90 GiB 4000.65 GB)
     Array Size : 11720658432 (11177.69 GiB 12001.95 GB)
  Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=688 sectors
          State : clean
    Device UUID : b835def0:00fd62dd:13e2a265:20a3c513

Internal Bitmap : 8 sectors from superblock
    Update Time : Fri Jun 18 11:14:48 2021
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : 2613bc81 - correct
         Events : 58428

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x9
     Array UUID : 6ee9d50a:6f215cef:9e5de359:d806fd4c
           Name : pi:0  (local to host pi)
  Creation Time : Mon Nov  2 23:52:14 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7813772976 (3725.90 GiB 4000.65 GB)
     Array Size : 11720658432 (11177.69 GiB 12001.95 GB)
  Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=688 sectors
          State : clean
    Device UUID : a76d62e3:9bf9c38c:b4dcee8f:b58cd1e8

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Jan 16 11:44:06 2022
  Bad Block Log : 512 entries available at offset 24 sectors - bad blocks present.
       Checksum : c8695a47 - correct
         Events : 58460

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6ee9d50a:6f215cef:9e5de359:d806fd4c
           Name : pi:0  (local to host pi)
  Creation Time : Mon Nov  2 23:52:14 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7813772976 (3725.90 GiB 4000.65 GB)
     Array Size : 11720658432 (11177.69 GiB 12001.95 GB)
  Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=688 sectors
          State : active
    Device UUID : 4a1653ce:1b8beeda:c91d6a4b:576df662

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Jan 16 11:44:06 2022
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : 3724b61a - correct
         Events : 58460

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 6ee9d50a:6f215cef:9e5de359:d806fd4c
           Name : pi:0  (local to host pi)
  Creation Time : Mon Nov  2 23:52:14 2020
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 7813772976 (3725.90 GiB 4000.65 GB)
     Array Size : 11720658432 (11177.69 GiB 12001.95 GB)
  Used Dev Size : 7813772288 (3725.90 GiB 4000.65 GB)
    Data Offset : 264192 sectors
   Super Offset : 8 sectors
   Unused Space : before=264112 sectors, after=688 sectors
          State : active
    Device UUID : d2048f05:ac9196fe:30fc31ae:802be8a7

Internal Bitmap : 8 sectors from superblock
    Update Time : Sun Jan 16 11:44:06 2022
  Bad Block Log : 512 entries available at offset 24 sectors
       Checksum : 39c247c5 - correct
         Events : 58460

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : .AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde:
   MBR Magic : aa55
Partition[0] :       524288 sectors at         8192 (type 0c)
Partition[1] :    351119408 sectors at       532480 (type 83)

mdadm raid status

pi@pi:~ $ checkraidstatus
/dev/md0:
           Version : 1.2
     Creation Time : Mon Nov  2 23:52:14 2020
        Raid Level : raid5
        Array Size : 11720658432 (11177.69 GiB 12001.95 GB)
     Used Dev Size : 3906886144 (3725.90 GiB 4000.65 GB)
      Raid Devices : 4
     Total Devices : 3
       Persistence : Superblock is persistent

     Intent Bitmap : Internal

       Update Time : Sun Jan 16 11:44:06 2022
             State : clean, degraded
    Active Devices : 3
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 0

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : bitmap

              Name : pi:0  (local to host pi)
              UUID : 6ee9d50a:6f215cef:9e5de359:d806fd4c
            Events : 58460

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       1       8       16        1      active sync   /dev/sdb
       2       8       32        2      active sync   /dev/sdc
       4       8       48        3      active sync   /dev/sdd
pi@pi:~ $ checkraidbuildstatus
pi@pi:~ $ cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active (auto-read-only) raid5 sdb[1] sdd[4] sdc[2]
      11720658432 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/3] [_UUU]
      bitmap: 3/30 pages [12KB], 65536KB chunk

unused devices: <none>

Any help would be greatly appreciated

2

It looks like sda has all the raid info still, but is not included in the actual running array?

I’ve not used mdadm, but have you tried adding sda back?
Not just a general scan for active arrays, and assembling arrays from active devices, but specifically re-adding sda back to the array?

I could be wrong, just going off the chart:

If sda is the device you used.

Does mdadm allow adding devices by device ID?
Because the command might be simple, like (NOT ACTUAL COMMAND) mdadm arryname replace /dev/sda /dev/disk/by-id/ata-Seagate-2937477npe

Then it might be easier to spot in future which serial number is bad, rather than the changing /dev/sdX?

3

when i try to add it I get a failed to write metadata error:

sudo mdadm /dev/md0 --add /dev/sda
mdadm: Failed to write metadata to /dev/sda
4

Not sure about the metadata, and not sure if you already fixed the array, but cam across a blog Replacing a failed drive in an mdadm array on Ubuntu – Miklix which seemed to insinuate “removing” the failed drive, then adding it.

They also talk about messing with the partition to avoid conflicts when adding it back?

And, bear in mind I could well be wrong; sda might never have been in the array, and/or the drive letters could have become jumbled around since last boot.

don’t want to put you boot drive in the array in error…

2 Likes
5

Let me try this. I’m pretty sure I had done something similar where I tried to remove and add the drive. That is where the entire raid started failing. Till before I was at least able to access my data. As of now I’m not able to mount the raid /dev/md0
I’ll get back on the result of following that link

6

soooo… i give up. Looks like i screwed up the raid big time. Tried your link and a host of other solutions. Have no option but to start from scratch !
Thanks for your help. Really appreciate it

2 Likes
7

You sure gave it a go.

I’m afraid it’s been years since I played with mdadm, and I didn’t actually have any trouble with it, so I’m sorry I couldn’t help.

I am surprised the sda drive could not be added back after a format, but I guess there must be some reason.

I run ZFS arrays, and when a dive gets kicked, one has to “replace” the virtual record it leaves behind, with the drive that got kicked out.
Shame mdadm does not work the same.

I hope you don’t give up on raid all together, but understand if it is too troublesome

1 Like
8

I’d have loved to try zfs but I’m working on a measly rpi 4, 8GB model. Took up raid so I could learn on the go but am a bit fearful now.
To be fair to raid, it was working when sda got removed. I just screwed it up trying to get it re-added. I should’ve just removed it, formatted it and re-added it.
Main reason why I didn’t do it was that rebuilding with a new 4tb takes a really long time on my pi when there is some data already in the raid. I guess I’m limited with the performance of the pi I suppose?

2 Likes
9

Fair enough.

I had a 4tb pool running over 2 drives on a pi4 4gb, one just have to lower the arc if one wants do do other stuff on the device.

But mdadm should be built in, so should have better compatibility across distro’s.

It;s a moot point, as the damage is done now.

I hear you on the desire to avoid starting from scratch, though the pi4 is not Bad on bandwidth, it’s just a lot lower than a desktop would have, and yeah, rebuilding a 4tb drive would take a while…

1 Like
10

thinking of giving raid 5 another go. is there any alternative you could suggest? my main usecase is; I want to be able to absorb at least 1 drive failure. my load [read/write] on the server is not going to be high but it will have a large amount of data. as of now I haven’t put any personal data on the earlier drive since I was still testing it out but would something like btrfs work better? All use cases I see are with regards to btrfs on root drive; which would mean my sdcard or the ssd I plan to use. All my data would be in a sata to usb enclosure which houses 4x4tb drives as of now

2 Likes
11

To be honest, I think mdadm is probably still the best use case.
If you like, you can format the filesystem on top of that (/dev/md0) as btrfs to take advantage of a lot of the features, snapshots and such.

There is other raid sets and file systems like lvm and such, but I have not tried them. And ZFS, which I love, but can get complicated (but need not be)

I run mostly ubuntu, and use zfs, but it’s not Better, and would mean a whole new chain, with lessons to learn, and a different set of problems.

But then again, I have backups of my data, even just the disposable media, like my dvd movie rips.

BTRFS is supposed to be able to create a raid5 array, but there are potential write hole issues with it, where it might not recover from interrupted power / unclean shutdowns, and a pi is Not stable enough for that, in my opinion.

I mean, I have had pi’s running non stop for like 4 months with no issue, but it does not take a lot for it to fall over.

So overall I would recommend carrying on with mdadm, and using this as a learning opportunity; You found the fault, you tried a couple things, and you think you’ve learned from it for the future.

more important than a raid that is bulletproof, is a backup that can be recovered from when a bigger bullet comes along and inevitably destroys the array again.

I am only using about a third of the space as I could, because I have a load of backups, and spares.
even a single 12tb USB drive could be a backup to a 4x4tb raid. but I am aware that is not a small chunk of change. I would still save up for it / budget for it for future.

Doesn’t help much I’m afraid /s but mostly don;t get scared off for one bad experience, raid is uptime, not a backup- raid should keep the system running while you make a backup and replace failed drives…

Just a fun tip, I pretty much use USB thumb/flash sticks as the root drives for my pi4’s; They are not much more expensive than sdcards, but are a bit more performant, and with log2ram on each system, I think they might last a bit longer.

2 Likes
12

thanks for the confidence booster. I think I needed it :slight_smile: … Started the raid5 creation. Considering it’s going on at about 57 MBps should take another day for it to complete.

As for the sdcard usage. I have a spare ssd and a sata to usb3 adapter. Before this I had actually migrated to the ssd. It’s just… not clean ! Too many wires. So I might (will mostly) migrate to that ssd later but for now I think I can make do with the sdcard.

will take a look at btrfs over raid option.

going forward, that’s the plan. I just need to figure out how to use things like rsync or maybe syncthing to have if not hourly at least nightly backups

1 Like