I’ve tried to do as much troubleshooting as i can but nothing has worked so far so I thought I should try posting here.
I am running Ubuntu Server with x4 8TB drives set up in RAID 10. Originally configured in mdadm when I first installed the OS. Everything was going fine until I tried to add x4 more 8TB drives to the array. I thought I did it correctly as it took a couple of days to finish and it reported that it went successfully. However /dev/md0 wasn’t using the extra space. I can’t remember exactly what i did to it but my Ubuntu install became corrupted and kept booting into recovery mode. Now whenever I try to start my array it keeps spitting out errors about no superblocks and not enough drives to start the array. I apologise if I don’t provide enough information I’m just lost on what to do.
ubuntu@ubuntu:~$ sudo mdadm --assemble --scan --force --verbose
mdadm: looking for devices for /dev/md/0
mdadm: No super block found on /dev/loop8 (Expected magic a92b4efc, got d5ab4258)
mdadm: no RAID superblock on /dev/loop8
mdadm: No super block found on /dev/dm-0 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/dm-0
mdadm: No super block found on 8:130 (Expected magic a92b4efc, got 07020701)
mdadm: no RAID superblock on 8:130
mdadm: No super block found on /dev/sdi1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdi1
mdadm: No super block found on /dev/sdi (Expected magic a92b4efc, got e86a9721)
mdadm: no RAID superblock on /dev/sdi
mdadm: No super block found on /dev/sdf1 (Expected magic a92b4efc, got 00000831)
mdadm: no RAID superblock on /dev/sdf1
mdadm: No super block found on /dev/sdf (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdf
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: No super block found on /dev/sdc (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdc
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/nvme1n1p3 (Expected magic a92b4efc, got 0000043c)
mdadm: no RAID superblock on /dev/nvme1n1p3
mdadm: No super block found on /dev/nvme1n1p2 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/nvme1n1p2
mdadm: No super block found on /dev/nvme1n1p1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/nvme1n1p1
mdadm: No super block found on /dev/nvme1n1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/nvme1n1
mdadm: No super block found on /dev/nvme0n1p4 (Expected magic a92b4efc, got e9ffc701)
mdadm: no RAID superblock on /dev/nvme0n1p4
mdadm: No super block found on /dev/nvme0n1p3 (Expected magic a92b4efc, got e30eccdf)
mdadm: no RAID superblock on /dev/nvme0n1p3
mdadm: No super block found on /dev/nvme0n1p2 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/nvme0n1p2
mdadm: No super block found on /dev/nvme0n1p1 (Expected magic a92b4efc, got 65a7ad5a)
mdadm: no RAID superblock on /dev/nvme0n1p1
mdadm: No super block found on /dev/nvme0n1 (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/nvme0n1
mdadm: No super block found on /dev/loop7 (Expected magic a92b4efc, got a6c8346c)
mdadm: no RAID superblock on /dev/loop7
mdadm: No super block found on /dev/loop6 (Expected magic a92b4efc, got 1a090af0)
mdadm: no RAID superblock on /dev/loop6
mdadm: No super block found on /dev/loop5 (Expected magic a92b4efc, got 3e22646e)
mdadm: no RAID superblock on /dev/loop5
mdadm: No super block found on /dev/loop4 (Expected magic a92b4efc, got c4b43b1a)
mdadm: no RAID superblock on /dev/loop4
mdadm: No super block found on /dev/loop3 (Expected magic a92b4efc, got 1bd7e597)
mdadm: no RAID superblock on /dev/loop3
mdadm: No super block found on /dev/loop2 (Expected magic a92b4efc, got 3a23b8f9)
mdadm: no RAID superblock on /dev/loop2
mdadm: /dev/loop1 is too small for md: size is 8 sectors.
mdadm: no RAID superblock on /dev/loop1
mdadm: No super block found on /dev/loop0 (Expected magic a92b4efc, got feedffbf)
mdadm: no RAID superblock on /dev/loop0
mdadm: /dev/sdj is identified as a member of /dev/md/0, slot 4.
mdadm: /dev/sdh is identified as a member of /dev/md/0, slot 5.
mdadm: /dev/sdg is identified as a member of /dev/md/0, slot 6.
mdadm: /dev/sde is identified as a member of /dev/md/0, slot 7.
mdadm: no uptodate device for slot 0 of /dev/md/0
mdadm: no uptodate device for slot 1 of /dev/md/0
mdadm: no uptodate device for slot 2 of /dev/md/0
mdadm: no uptodate device for slot 3 of /dev/md/0
mdadm: added /dev/sdh to /dev/md/0 as 5
mdadm: added /dev/sdg to /dev/md/0 as 6
mdadm: added /dev/sde to /dev/md/0 as 7
mdadm: added /dev/sdj to /dev/md/0 as 4
mdadm: /dev/md/0 assembled from 4 drives - not enough to start the array.
ubuntu@ubuntu:~$ sudo mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Raid Level : raid10
Total Devices : 4
Persistence : Superblock is persistent
State : inactive
Working Devices : 4
Name : ubuntu-server:0
UUID : ad6b3838:2414b1d5:ec5ddd26:af9c216e
Events : 134880
Number Major Minor RaidDevice
- 8 64 - /dev/sde
- 8 144 - /dev/sdj
- 8 112 - /dev/sdh
- 8 96 - /dev/sdg
ubuntu@ubuntu:~$ sudo mdadm --examine /dev/sd[ejhg]
/dev/sde:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ad6b3838:2414b1d5:ec5ddd26:af9c216e
Name : ubuntu-server:0
Creation Time : Mon Apr 25 03:30:35 2022
Raid Level : raid10
Raid Devices : 8
Avail Dev Size : 15627790000 sectors (7.28 TiB 8.00 TB)
Array Size : 31255578624 KiB (29.11 TiB 32.01 TB)
Used Dev Size : 15627789312 sectors (7.28 TiB 8.00 TB)
Data Offset : 263168 sectors
Super Offset : 8 sectors
Unused Space : before=263080 sectors, after=688 sectors
State : clean
Device UUID : 7e855b7b:deea5506:96349577:2b25d081
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Oct 23 08:53:58 2022
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 623d147b - correct
Events : 134880
Layout : near=2
Chunk Size : 512K
Device Role : Active device 7
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdg:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ad6b3838:2414b1d5:ec5ddd26:af9c216e
Name : ubuntu-server:0
Creation Time : Mon Apr 25 03:30:35 2022
Raid Level : raid10
Raid Devices : 8
Avail Dev Size : 15627790000 sectors (7.28 TiB 8.00 TB)
Array Size : 31255578624 KiB (29.11 TiB 32.01 TB)
Used Dev Size : 15627789312 sectors (7.28 TiB 8.00 TB)
Data Offset : 263168 sectors
Super Offset : 8 sectors
Unused Space : before=263080 sectors, after=688 sectors
State : clean
Device UUID : b50b1791:f6998cd1:20ca9527:f00624fd
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Oct 23 08:53:58 2022
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 6e83c11b - correct
Events : 134880
Layout : near=2
Chunk Size : 512K
Device Role : Active device 6
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdh:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ad6b3838:2414b1d5:ec5ddd26:af9c216e
Name : ubuntu-server:0
Creation Time : Mon Apr 25 03:30:35 2022
Raid Level : raid10
Raid Devices : 8
Avail Dev Size : 15627790000 sectors (7.28 TiB 8.00 TB)
Array Size : 31255578624 KiB (29.11 TiB 32.01 TB)
Used Dev Size : 15627789312 sectors (7.28 TiB 8.00 TB)
Data Offset : 263168 sectors
Super Offset : 8 sectors
Unused Space : before=263080 sectors, after=688 sectors
State : clean
Device UUID : dd8a9053:9d0a5076:220829ff:76c4294c
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Oct 23 08:53:58 2022
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : fc59ac72 - correct
Events : 134880
Layout : near=2
Chunk Size : 512K
Device Role : Active device 5
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdj:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : ad6b3838:2414b1d5:ec5ddd26:af9c216e
Name : ubuntu-server:0
Creation Time : Mon Apr 25 03:30:35 2022
Raid Level : raid10
Raid Devices : 8
Avail Dev Size : 15627790000 sectors (7.28 TiB 8.00 TB)
Array Size : 31255578624 KiB (29.11 TiB 32.01 TB)
Used Dev Size : 15627789312 sectors (7.28 TiB 8.00 TB)
Data Offset : 263168 sectors
Super Offset : 8 sectors
Unused Space : before=263080 sectors, after=688 sectors
State : clean
Device UUID : 6d03c728:18db42e5:b9e21779:17846ff2
Internal Bitmap : 8 sectors from superblock
Update Time : Sun Oct 23 08:53:58 2022
Bad Block Log : 512 entries available at offset 72 sectors
Checksum : 60b78fb7 - correct
Events : 134880
Layout : near=2
Chunk Size : 512K
Device Role : Active device 4
Array State : AAAAAAAA ('A' == active, '.' == missing, 'R' == replacing)
ubuntu@ubuntu:~$ sudo fdisk -l
Disk /dev/loop0: 2.33 GiB, 2502324224 bytes, 4887352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop1: 4 KiB, 4096 bytes, 8 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop2: 61.89 MiB, 64901120 bytes, 126760 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop3: 248.76 MiB, 260841472 bytes, 509456 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop4: 155.63 MiB, 163188736 bytes, 318728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop5: 45.86 MiB, 48087040 bytes, 93920 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop6: 43.63 MiB, 45748224 bytes, 89352 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop7: 284 KiB, 290816 bytes, 568 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/nvme0n1: 232.89 GiB, 250059350016 bytes, 488397168 sectors
Disk model: Samsung SSD 970 EVO Plus 250GB
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 25219762-95A8-4EF4-B80E-4B0E7B825FA6
Device Start End Sectors Size Type
/dev/nvme0n1p1 4096 6143 2048 1M BIOS boot
/dev/nvme0n1p2 6144 1054719 1048576 512M EFI System
/dev/nvme0n1p3 34609152 488397134 453787983 216.4G Solaris /usr & Apple ZFS
/dev/nvme0n1p4 1054720 34609151 33554432 16G Linux swap
Partition table entries are not in disk order.
Disk /dev/nvme1n1: 1.82 TiB, 2000398934016 bytes, 3907029168 sectors
Disk model: CT2000P1SSD8
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 7DBD0048-CC6F-47B7-9D1D-DFF8F0EC4778
Device Start End Sectors Size Type
/dev/nvme1n1p1 2048 2203647 2201600 1G EFI System
/dev/nvme1n1p2 2203648 10592255 8388608 4G Linux swap
/dev/nvme1n1p3 10592256 3907028991 3896436736 1.8T Linux filesystem
Disk /dev/sda: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 0D138220-2859-4650-93BF-995597CF7432
Disk /dev/sdb: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: DF32EDEF-2811-4AA9-8907-B458BDEFFCB3
Disk /dev/sdc: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9B38A391-A114-4F19-ABAF-1D51B203A413
Disk /dev/sdd: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-2M21
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: F8A132FC-7182-4F45-AA6C-249FC54B1DB1
The primary GPT table is corrupt, but the backup appears OK, so that will be used.
Disk /dev/sde: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-3CP1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 9E44DF4B-EEBE-C041-9165-AE8A48979F0E
Device Start End Sectors Size Type
/dev/sde1 2048 15628053134 15628051087 7.3T Linux filesystem
Disk /dev/sdf: 16.37 TiB, 18000207937536 bytes, 35156656128 sectors
Disk model: Generic
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 181FFD43-67EA-442A-B8FF-0F7340AE38CA
Device Start End Sectors Size Type
/dev/sdf1 2048 35156654079 35156652032 16.4T Linux filesystem
The primary GPT table is corrupt, but the backup appears OK, so that will be used.
Disk /dev/sdg: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-3CP1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 86748330-EA1C-8A4C-8A81-E275EB752D85
Device Start End Sectors Size Type
/dev/sdg1 2048 15628053134 15628051087 7.3T Linux filesystem
The primary GPT table is corrupt, but the backup appears OK, so that will be used.
Disk /dev/sdh: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-3CP1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: BF77BC62-B1F6-254C-BFA5-19265B42CDAF
Device Start End Sectors Size Type
/dev/sdh1 2048 15628053134 15628051087 7.3T Linux filesystem
Disk /dev/sdi: 57.62 GiB, 61865984000 bytes, 120832000 sectors
Disk model: USB Flash Drive
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xc8f094fe
Device Boot Start End Sectors Size Id Type
/dev/sdi1 * 2048 120766463 120764416 57.6G 7 HPFS/NTFS/exFAT
/dev/sdi2 120766464 120831999 65536 32M ef EFI (FAT-12/16/32)
The primary GPT table is corrupt, but the backup appears OK, so that will be used.
Disk /dev/sdj: 7.28 TiB, 8001563222016 bytes, 15628053168 sectors
Disk model: ST8000VN004-3CP1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: 517F2D59-9B0E-4B46-82CF-DCC530CB6564
Device Start End Sectors Size Type
/dev/sdj1 2048 15628053134 15628051087 7.3T Linux filesystem
Disk /dev/mapper/ventoy: 3.4 GiB, 3654957056 bytes, 7138588 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: A09DB2B8-B5F6-43AE-AFB3-91E0A90189A1
Device Start End Sectors Size Type
/dev/mapper/ventoy-part1 64 7129427 7129364 3.4G Microsoft basic data
/dev/mapper/ventoy-part2 7129428 7137923 8496 4.1M EFI System
/dev/mapper/ventoy-part3 7137924 7138523 600 300K Microsoft basic data
Disk /dev/loop8: 81.26 MiB, 85209088 bytes, 166424 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
If anyone requests any more information I’ll gladly do so. I’m grabbing this info from a live instance of Ubuntu. I just wanna get my array back up so I can pull all my data off it for a backup (which in hindsight I should’ve done before I started messing with my array).
