RAID dead please help

Operating Systems & Open Source Linux
1

i have a mdadm raid on md0 that should be running from 4 disks

im seeing missing superblocks on the ext4s mdadm is running on. im not sure if i should repair this or try to repair mdadm on the partition.

sudo cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : inactive sdb1[4] sdc1[7] sda1[6]
      8790408022 blocks super 1.2
       
unused devices: <none>

sudo mdadm -E /dev/sd[abcde]
/dev/sda:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdd:
   MBR Magic : aa55
Partition[0] :    234438656 sectors at         2048 (type 8e)
/dev/sde:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)

EDIT:1

sudo mdadm --examine --scan
ARRAY /dev/md/0  metadata=1.2 UUID=1a63917c:d8c53d78:90f31fa3:3ef66b20 name=wrath:0

fdisk -l lists my partitions, so thats good?

EDIT:2

sudo mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md/0
mdadm: No super block found on /dev/dm-1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/dm-1
mdadm: No super block found on /dev/dm-0 (Expected magic a92b4efc, got 0000040f)
mdadm: no RAID superblock on /dev/dm-0
mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sde
mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 7e51852e)
mdadm: no RAID superblock on /dev/sdd1
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdc1 is busy - skipping
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: /dev/sdb1 is busy - skipping
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: /dev/sda1 is busy - skipping
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/loop3 (Expected magic a92b4efc, got 94557b95)
mdadm: no RAID superblock on /dev/loop3
mdadm: No super block found on /dev/loop2 (Expected magic a92b4efc, got fd6d4056)
mdadm: no RAID superblock on /dev/loop2
mdadm: No super block found on /dev/loop1 (Expected magic a92b4efc, got fd6d4056)
mdadm: no RAID superblock on /dev/loop1
mdadm: No super block found on /dev/loop0 (Expected magic a92b4efc, got fd6d4056)
mdadm: no RAID superblock on /dev/loop0
mdadm: Found some drive for an array that is already active: /dev/md/0
mdadm: giving up.

EDIT:3

sudo fsck.ext4 /dev/sda1
e2fsck 1.44.1 (24-Mar-2018)
/dev/sda1 is in use.

sda1 is not mounted

EDIT:4
sudo mke2fs -n /dev/sdb1
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdb1 contains a linux_raid_member file system labelled ‘wrath:0’
Proceed anyway? (y,N) y
Creating filesystem with 732566385 4k blocks and 183148544 inodes
Filesystem UUID: f49f6666-6cd5-4ba4-9493-553f3a6452fa
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
102400000, 214990848, 512000000, 550731776, 644972544

koda@wrath:~$ sudo mke2fs -n /dev/sdc1
mke2fs 1.44.1 (24-Mar-2018)
/dev/sdc1 contains a linux_raid_member file system labelled ‘wrath:0’
Proceed anyway? (y,N) y
/dev/sdc1 is apparently in use by the system; will not make a filesystem here!

in use by system is really in my way, still not mounted

please, any help, thank you, im not sure what to do

2 
sudo mdadm --stop /dev/md0
sudo mdmadm --assemble --force /dev/md0 /dev/sd[abc]1
1 Like
3

chance of damage or will it check first?

4

As long as you dont fsck or repart the drives it should be a safe operation.

5

not sure from the output if sdd is supposed to be part of the array, if so, update the assemble command appropriately

6

sudo mdadm --assemble --force /dev/md0 /dev/sd[bcde]1
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdd1 is busy - skipping
mdadm: /dev/sde1 is busy - skipping
mdadm: Found some drive for an array that is already active: /dev/md/0
mdadm: giving up.

7

they are currently on bcde

8

output of lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT ?

9 
NAME                   SIZE FSTYPE            TYPE MOUNTPOINT
loop0                193.1M squashfs          loop /snap/acestreamplayer/7
loop1                 87.9M squashfs          loop /snap/core/5548
loop2                 87.9M squashfs          loop /snap/core/5662
loop3                 87.9M squashfs          loop /snap/core/5742
sda                  111.8G                   disk 
└─sda1               111.8G LVM2_member       part 
  ├─wrath--vg-root   107.8G ext4              lvm  /
  └─wrath--vg-swap_1     4G swap              lvm  [SWAP]
sdb                    3.7T                   disk 
└─sdb1                 2.7T linux_raid_member part 
sdc                    3.7T                   disk 
└─sdc1                 2.7T linux_raid_member part 
sdd                    3.7T                   disk 
└─sdd1                 2.7T linux_raid_member part 
sde                    2.7T                   disk 
└─sde1                 2.7T linux_raid_member part

EDIT: removed error

10

im not crying

11

Did you --stop the array before trying --assemble?

What does “mdadm --detail /dev/md0” output?

12 
sudo mdadm --detail /dev/md0
/dev/md0:
           Version : 1.2
     Creation Time : Mon Nov 20 18:37:58 2017
        Raid Level : raid5
     Used Dev Size : 18446744073709551615
      Raid Devices : 4
     Total Devices : 3
       Persistence : Superblock is persistent

       Update Time : Wed Oct 31 19:22:34 2018
             State : active, FAILED, Not Started 
    Active Devices : 2
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 1

            Layout : left-symmetric
        Chunk Size : 512K

Consistency Policy : unknown

              Name : wrath:0  (local to host wrath)
              UUID : 1a63917c:d8c53d78:90f31fa3:3ef66b20
            Events : 28169

    Number   Major   Minor   RaidDevice State
       -       0        0        0      removed
       -       0        0        1      removed
       -       0        0        2      removed
       -       0        0        3      removed

       -       8       65        3      spare rebuilding   /dev/sde1
       -       8       49        1      sync   /dev/sdd1
       -       8       33        0      sync   /dev/sdc1
13

no, im new to broken raids, first time ive had an issue

14

so what’s the output of /proc/mdstat now?

15 
Personalities : [raid6] [raid5] [raid4] [linear] [multipath] [raid0] [raid1] [raid10] 
md0 : inactive sde1[4] sdd1[7] sdc1[6]
      8790408022 blocks super 1.2
       
unused devices: <none>
16

And at this point mdadm -A /dev/md0 /dev/sd[cde]1 doesn’t activate the array?

17 
sudo mdadm -A /dev/md0 /dev/sd[bcde]1
mdadm: /dev/sdc1 is busy - skipping
mdadm: /dev/sdd1 is busy - skipping
mdadm: /dev/sde1 is busy - skipping
mdadm: Found some drive for an array that is already active: /dev/md/0
mdadm: giving up.

no it tells me that some of the drives are busy

18 
sudo mdadm --assemble --scan -v
mdadm: looking for devices for /dev/md/0
mdadm: No super block found on /dev/dm-1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/dm-1
mdadm: No super block found on /dev/dm-0 (Expected magic a92b4efc, got 0000040f)
mdadm: no RAID superblock on /dev/dm-0
mdadm: /dev/sde1 is busy - skipping
mdadm: No super block found on /dev/sde (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sde
mdadm: /dev/sdd1 is busy - skipping
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdc1 is busy - skipping
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda1 (Expected magic a92b4efc, got 7e51852e)
mdadm: no RAID superblock on /dev/sda1
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got a0ddb71c)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/loop3 (Expected magic a92b4efc, got fd6d4056)
mdadm: no RAID superblock on /dev/loop3
mdadm: No super block found on /dev/loop2 (Expected magic a92b4efc, got fd6d4056)
mdadm: no RAID superblock on /dev/loop2
mdadm: No super block found on /dev/loop1 (Expected magic a92b4efc, got fd6d4056)
mdadm: no RAID superblock on /dev/loop1
mdadm: No super block found on /dev/loop0 (Expected magic a92b4efc, got 94557b95)
mdadm: no RAID superblock on /dev/loop0
mdadm: Found some drive for an array that is already active: /dev/md/0
mdadm: giving up.
19

What do the drives look like individually? “mdadm --examine /dev/sdb1” … ?

20 
sudo mdadm --examine /dev/sd[bcde]1
/dev/sdb1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 1a63917c:d8c53d78:90f31fa3:3ef66b20
           Name : wrath:0  (local to host wrath)
  Creation Time : Mon Nov 20 18:37:58 2017
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860272015 (2794.40 GiB 3000.46 GB)
     Array Size : 8790406656 (8383.19 GiB 9001.38 GB)
  Used Dev Size : 5860271104 (2794.40 GiB 3000.46 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=911 sectors
          State : clean
    Device UUID : 37216ffb:21c8dc66:ff9349bb:36b56f62

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Oct 31 18:59:19 2018
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 810adfab - correct
         Events : 27374

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 1a63917c:d8c53d78:90f31fa3:3ef66b20
           Name : wrath:0  (local to host wrath)
  Creation Time : Mon Nov 20 18:37:58 2017
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860272015 (2794.40 GiB 3000.46 GB)
     Array Size : 8790406656 (8383.19 GiB 9001.38 GB)
  Used Dev Size : 5860271104 (2794.40 GiB 3000.46 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=911 sectors
          State : clean
    Device UUID : fdff18a4:05ce15be:b8999631:e71244f3

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Oct 31 19:22:34 2018
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 880c304f - correct
         Events : 28169

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : AA.A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 1a63917c:d8c53d78:90f31fa3:3ef66b20
           Name : wrath:0  (local to host wrath)
  Creation Time : Mon Nov 20 18:37:58 2017
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860272015 (2794.40 GiB 3000.46 GB)
     Array Size : 8790406656 (8383.19 GiB 9001.38 GB)
  Used Dev Size : 5860271104 (2794.40 GiB 3000.46 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
   Unused Space : before=258992 sectors, after=911 sectors
          State : clean
    Device UUID : 0700751a:77695db8:d5c753d4:80eaa279

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Oct 31 19:22:34 2018
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : 21cbd182 - correct
         Events : 28169

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sde1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x8b
     Array UUID : 1a63917c:d8c53d78:90f31fa3:3ef66b20
           Name : wrath:0  (local to host wrath)
  Creation Time : Mon Nov 20 18:37:58 2017
     Raid Level : raid5
   Raid Devices : 4

 Avail Dev Size : 5860272015 (2794.40 GiB 3000.46 GB)
     Array Size : 8790406656 (8383.19 GiB 9001.38 GB)
  Used Dev Size : 5860271104 (2794.40 GiB 3000.46 GB)
    Data Offset : 259072 sectors
   Super Offset : 8 sectors
Recovery Offset : 5456 sectors
   Unused Space : before=258992 sectors, after=911 sectors
          State : clean
    Device UUID : f7053583:52191f7a:f90eee36:1e57fb53

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed Oct 31 19:22:34 2018
  Bad Block Log : 512 entries available at offset 16 sectors - bad blocks present.
       Checksum : 894050e5 - correct
         Events : 28169

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AA.A ('A' == active, '.' == missing, 'R' == replacing)