Advice on mdadm RAID6 setup?

Software & Operating Systems Linux
1

I recently set up a raid6. However, various instructions found online take different ways about it.

Some don’t make partitions. Some don’t set flags on parted. Etc…

I largely followed this guide: How To Create RAID Arrays with mdadm on Ubuntu 18.04 | DigitalOcean

And ended up with this:

~$ lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT
sda                             5.47T                   disk
└─sda1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data
sdb                             5.47T                   disk
└─sdb1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data
sdc                             5.47T                   disk
└─sdc1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data
sdd                             5.47T                   disk
└─sdd1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data
sde                             5.47T                   disk
└─sde1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data
sdf                             5.47T                   disk
└─sdf1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data
sdg                             5.47T                   disk
└─sdg1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data
sdh                             5.47T                   disk
└─sdh1                          5.47T linux_raid_member part
  └─md0                         43.8T ext4              raid6 /data

And I don’t know if this is good or not.

Printing one of the disks gives:

(parted) print
Model: ATA WDC <REMOVED> (scsi)
Disk /dev/sdh: 6.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags:

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  6.0TB   6.0TB                      raid

And in fdisk:

Command (m for help): print
Disk /dev/sdh: 5.47 TiB, 6000900661248 bytes, 11719409651 sectors
Disk model: WDC <REMOVED>
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: CA0B5B87-6697-4350-A772-D48444119ABC

Device     Start         End     Sectors  Size Type
/dev/sdh1   2048 11719409651 11719409651 5.47T Linux RAID

I see other guides saying to give each disk a primary partition. Is this needed if they’re going to be raid?

Honestly, this is all way too confusing, with all these guides that each leaves different things out…

Any advice would be appreciated. Is this the way to do it? Am I missing something crucial by not setting a filesystem? Something else?

I just wanted to test using mdadm, never done it before… and there’s just so much conflicting info out there.

2

Both methods are fine. Given that you’re testing, you can try both and see what difference it makes. For easy RAID setup, install Webmin and manage the pool via your browser. Do note it needs full root access, so sudo is a non-starter as it doesn’t elevate permissions sufficiently.

3

It looks fine.

Using partitions instead of the entire disk means OS’s have a better understanding of the data there, and you have a unique partition ID (with GPT) for each one which helps identification with tools when the device IDs change. For example, if you use the entire disk as MD RAID, and - heaven forbid - put that disk in a Windows machine, it’ll happily ask you to “initialize” the raw disk with unknown data format on it, but using a partition it’ll leave it alone.

It doesn’t make any difference to MD, because there is an additional GUID in the metadata of each RAID component to assemble the group.

“primary partition” is MBR terminology - not meaningful for GPT. I’m guessing you mean an additional partition before the partition used for MD RAID. Makes no difference. I personally put a ~200MB ESP partition on every drive, even if I intend to use it as data only, as it gives me an option to easily make it bootable if I need to in the future, and it doesn’t take much space to do that.

At the end of the day, MD just takes multiple block devices to create a new block device, it doesn’t care where they come from, but it can make a difference to the tools/OS’s outside of MD.

2 Likes
4

Thanks for the advice ya’ll.

I wrote down the steps I used, comes down to something like this:

  1. Check your disks

lsblk -o NAME,SIZE,FSTYPE,TYPE,MOUNTPOINT

  1. Check if disks already have raid or not. Example:

sudo mdadm -E /dev/sd[a-h]

  1. Create a raid partition on each disk. Example. Use on all disks you want to include:
parted -a optimal /dev/sda

mklabel GPT

set 1 raid on

quit
  1. Make each disk a raid partition

Press 'n' for creating new partition.

Then choose 'P' for Primary partition.

Next choose the partition number as 1.

Define the default value by just pressing two times Enter key.

Next press 'P' to print the defined partition.

Press 'L' to list all available types.

Type 't' to choose the partitions.

Choose 'fd' for Linux raid auto and press Enter to apply.

Then again use 'P' to print the changes what we have made.

Use 'w' to write the changes
  1. Create the raid. Example using raid6 and 8 drives.

sudo mdadm --create --verbose /dev/md0 --level=6 --raid-devices=8 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdf1 /dev/sdg1 /dev/sdh1

  1. Make the created device ext4

sudo mkfs.ext4 -v -m 1 -b 4096 -E stride=128,stripe-width=768 /dev/md0

Use a guide like this: Calculating filesystem stride_size and stripe_width for best performance under RAID - Bugbear Thoughts

Or a tool like this: mkfs.ext3 RAID stride calculation

To generate your settings. Depending on your chunk size, you might need a non-default stripe width.

  1. Wait for the raid to finish building

cat /proc/mdstat

or

sudo mdadm --detail /dev/md0

  1. Create mount directory(location of your choosing)

sudo mkdir -p /data

  1. Add result to your conf, so the array gets reassembled on boot

sudo mdadm --detail --scan | sudo tee -a /etc/mdadm/mdadm.conf

  1. Make sure the array is available during early boot process

sudo update-initramfs -u

  1. Set the mount in /etc/fstab (these are just the options I used)

/dev/md0 /data ext4 lazytime,async,nodev,noexec,nosuid,stripe=768,rw,nofail 0 0

No discard in my case, these aren’t SSDs.

  1. Perform the mount

sudo mount /data

1 Like