LVM RAID with SSD Cache Guide

LinuxMaster9 · November 7, 2019, 4:16am

* UPDATED GUIDE 2024 ****

LVM2 RAID6 with SSD Caching: A Complete Setup Guide

This guide walks you through setting up an LVM2 RAID 6 array with SSD caching, ensuring optimal performance and persistence after system reboots.

Key Terminology

PV (Physical Volume): Base storage devices (e.g., HDDs, SSDs).
VG (Volume Group): A pool of PVs used to create LVs.
LV (Logical Volume): The usable storage, similar to partitions.
RAID Types: LVM supports RAID levels 0, 1, 4, 5, 6, 10. RAID 6 offers dual parity for better fault tolerance.

This guide assumes:

8 HDDs for RAID 6.
2 SSDs for a mirrored (RAID 1) cache pool.
A Linux system with LVM2 installed.
/dev/sda is the OS
replace any [b-h] reference with your drive identifiers. Adjust instructions to fit your setup.

Step 1: Install Required Tools

Ensure LVM2 and thin-provisioning-tools are installed (required for cache persistence after reboot on Debian/Ubuntu):

sudo apt-get install lvm2 thin-provisioning-tools

Step 2: Prepare the Drives

Clean all drives by removing partitions or existing data:
```
sudo wipefs -a /dev/sd[b-h] /dev/nvme0n1 /dev/nvme1n1
```
Initialize the HDDs as physical volumes (PVs):
```
sudo pvcreate /dev/sd[b-h]
```
Verify PV creation:
```
sudo pvscan
```

Step 3: Create a Volume Group

Create a volume group (VG) using the HDDs:
```
sudo vgcreate Library /dev/sd[b-h]
```
Confirm the VG creation:
```
sudo vgs
```

Step 4: Create the RAID 6 Logical Volume

Create a RAID 6 logical volume (LV):
```
sudo lvcreate --type raid6 -i 6 -l 100%FREE -n LibraryVolume Library
```
- --type: RAID type.
- -i: Number of data stripes (HDDs minus parity drives).
- -l: Allocate all available space.
- -n: Name of the LV.
Verify the LV creation:
```
sudo lvs
```

Step 5: Add SSDs for Caching

Convert SSDs into PVs:

sudo pvcreate /dev/nvme0n1 /dev/nvme1n1

Extend the VG to include the SSDs:

sudo vgextend Library /dev/nvme0n1 /dev/nvme1n1

Step 6: Create a Cache Pool

Create a mirrored (RAID 1) cache LV:

sudo lvcreate --type raid1 -m 1 -L 225G -n cache1 Library /dev/nvme0n1 /dev/nvme1n1

Create the metadata LV for the cache:

sudo lvcreate --type raid1 -m 1 -L 225M -n cache1meta Library /dev/nvme0n1 /dev/nvme1n1

Combine the cache and metadata LVs into a cache pool:

sudo lvconvert --type cache-pool --poolmetadata Library/cache1meta Library/cache1

Attach the cache pool to the RAID 6 LV:
```
sudo lvconvert --type cache --cachepool Library/cache1 --cachemode writethrough Library/LibraryVolume
```
- writethrough: Writes data to both the cache and RAID simultaneously (default, safer).
- writeback: Writes data to the cache first for better performance (riskier).

Step 7: Format and Mount the Logical Volume

Format the LV with your desired filesystem:

sudo mkfs.ext4 /dev/Library/LibraryVolume

Mount the volume:

sudo mkdir /mnt/Library
sudo mount /dev/Library/LibraryVolume /mnt/Library

Add the LV to /etc/fstab for automatic mounting:

echo "/dev/Library/LibraryVolume /mnt/Library ext4 defaults 0 2" | sudo tee -a /etc/fstab

Step 8: Verify the Setup

Check the disk layout:
```
lsblk
```

Example output:

NAME            SIZE TYPE  MOUNTPOINT
...
LibraryVolume  21.9T lvm   /mnt/Library
...

Best Practices

Monitor your setup using:
```
sudo lvs
sudo vgs
sudo pvs
```
Backup critical data, especially when using writeback mode.
Use tools like iotop or iostat to monitor I/O performance.

By following this guide, you can set up an efficient and persistent LVM2 RAID 6 array with SSD caching, leveraging the latest tools and best practices for data safety and performance.

Here is an example of lsblk after I set it up all those years ago. You will notice that my array includes /dev/sda. This is because at the time my OS lived on /dev/sdi.

NAME                                        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

sda                                           8:0    0   3.7T  0 disk 
└─sda1                                        8:1    0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_0  253:0    0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_0 253:1    0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sdb                                           8:16   0   3.7T  0 disk 
└─sdb1                                        8:17   0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_1  253:2    0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_1 253:3    0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sdc                                           8:32   0   3.7T  0 disk 
└─sdc1                                        8:33   0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_2  253:4    0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_2 253:5    0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sdd                                           8:48   0   3.7T  0 disk 
└─sdd1                                        8:49   0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_3  253:6    0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_3 253:7    0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sde                                           8:64   0   3.7T  0 disk 
└─sde1                                        8:65   0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_4  253:8    0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_4 253:9    0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sdf                                           8:80   0   3.7T  0 disk 
└─sdf1                                        8:81   0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_5  253:10   0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_5 253:11   0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sdg                                           8:96   0   3.7T  0 disk 
└─sdg1                                        8:97   0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_6  253:12   0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_6 253:13   0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sdh                                           8:112  0   3.7T  0 disk 
└─sdh1                                        8:113  0   3.7T  0 part 
  ├─Library-LibraryVolumeLMV_corig_rmeta_7  253:14   0     4M  0 lvm  
  │ └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
  │   └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library
  └─Library-LibraryVolumeLMV_corig_rimage_7 253:15   0   3.7T  0 lvm  
    └─Library-LibraryVolumeLMV_corig        253:27   0  21.9T  0 lvm  
      └─Library-LibraryVolumeLMV            253:16   0  21.9T  0 lvm  /mnt/Library

sdi                                           8:128  0 223.6G  0 disk 
└─sdi1                                        8:129  0 223.6G  0 part /

nvme1n1                                     259:0    0 238.5G  0 disk 
├─Library-cache1_cdata_rmeta_1              253:19   0     4M  0 lvm  
│ └─Library-cache1_cdata                    253:21   0   225G  0 lvm  
│   └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library
├─Library-cache1_cdata_rimage_1             253:20   0   225G  0 lvm  
│ └─Library-cache1_cdata                    253:21   0   225G  0 lvm  
│   └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library
├─Library-cache1_cmeta_rmeta_1              253:24   0     4M  0 lvm  
│ └─Library-cache1_cmeta                    253:26   0   228M  0 lvm  
│   └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library
└─Library-cache1_cmeta_rimage_1             253:25   0   228M  0 lvm  
  └─Library-cache1_cmeta                    253:26   0   228M  0 lvm  
    └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library

nvme0n1                                     259:1    0 238.5G  0 disk 
├─Library-cache1_cdata_rmeta_0              253:17   0     4M  0 lvm  
│ └─Library-cache1_cdata                    253:21   0   225G  0 lvm  
│   └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library
├─Library-cache1_cdata_rimage_0             253:18   0   225G  0 lvm  
│ └─Library-cache1_cdata                    253:21   0   225G  0 lvm  
│   └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library
├─Library-cache1_cmeta_rmeta_0              253:22   0     4M  0 lvm  
│ └─Library-cache1_cmeta                    253:26   0   228M  0 lvm  
│   └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library
└─Library-cache1_cmeta_rimage_0             253:23   0   228M  0 lvm  
  └─Library-cache1_cmeta                    253:26   0   228M  0 lvm  
    └─Library-LibraryVolumeLMV              253:16   0  21.9T  0 lvm  /mnt/Library

zlynx · November 7, 2019, 9:07pm

Nitpick: You seem to switch between including /dev/sda and not including it. sda is part of your Library down at the end, but isn’t included in your setup instructions at the top.

vlycop · November 7, 2019, 10:23pm

Dumb question, but if i want to do the same with a ramdisk, should i search for write only cache ? or using write-through is enought ?

zlynx · November 7, 2019, 11:22pm

What sort of advantage do you plan to get from a ramdisk that you wouldn’t get from the regular Linux file cache?

LinuxMaster9 · November 8, 2019, 2:41am

This is because, Usually, /dev/sda is the root device. In this case, mine was /dev/sdi.

The point of this guide is for you to use lsblk to determine the device names to use BEFORE continuing the guide.

Also, the reference to /dev/sda in the beginning is to give an example of what you are looking for.

vlycop · November 8, 2019, 9:23am

I simply have to much ram (old video computer, 32gb) and no spare ssd since a recent magic smoke accident . I try to have a read cache for my iscsi steam library

daniel712 · March 18, 2021, 10:45pm

Hi Friends,
i am planning a new setup for my main machine on basis of lvmraid. I am using the Taichi X570 with a Matisse CPU. So i have 3 damn fast M.2 Gen4 SSDs and 8 SATA Conns.
First Plan was to create a RAID 5 with the 3 NVMEs for root and home and to bring in the SATA SSDs per on demand mount points (backup, archive, media, etc)
Could you estimate to what extend your setup, where i would use the 3 NVMEs only for cache and metadata, will reach the speed of the native RAID5 on the 3 NVMEs? I don’t have any “feeling” about that. So if you have some experience with your setup, you could help me a lot for a decision, even if it’s a rough guess only!
Thanks a lot and best regards from Wurzburg, Germany
Daniel

nx2l · March 18, 2021, 11:01pm

probably worth trying a few different setups to find what you like best.

Try root and home on their own SSD.

Then try the raid5 of them…

Test out performance on each… see what feels faster?

daniel712 · March 18, 2021, 11:11pm

Without testing i would guess that the RAID5 over the 3 NVMEs is the fastest solution. But with respect to cost/capacity i would be happy for a guess “how much” speed i loose with the NVME Cache + SATA Storae solution of this concept here…
For my RAID5 with NVMEs i would have 1TB capacity with 3 disks.
With the SATA + cache solutiion i would have 4TB with 6 disks + 2 small NVMEs.
The Costs would be comparable. So if i had an advice how fast the caching solution is, i would definitely build this. But if i loose lets say 50% speed, i would compromise the capacity and go for the superfast NVME RAID…

daniel712 · March 18, 2021, 11:26pm

Little bits of understanding:
The cache is writeback, so i have the full NVME Speed in writing, but i need a RAID1 Setup for the Cache. But no Problem, i could write files up to the cache Size, lets say 250GB with full speed. So not too much loss there…
What about the random reads? There is the main speedloss of the SATA+Cache solution, correct? For the root FS the Cache rate could be fairly acceptable, but on home… no clue… i think there will be not too much more than the SATA speed for the random reads…

JoeyD · April 15, 2021, 7:40am

How do you run the maintenance commands on the raid part of the cached-LVs? None of the usual commands for checking sync and such seem to work on the hidden LVs behind the cache. At least I can’t figure out how and I also haven’t been able to find anything online about it either.

I’m starting to suspect that I have to first turn off the cache and then do all the checks and maintenance steps. That would be a massive pain if I have to do that for every blown fuse or the occasional poweroutage.

LinuxMaster9 · April 24, 2021, 11:35pm

you should be able to run the scrub command just fine. I had no issues doing that.

LinuxMaster9 · April 24, 2021, 11:37pm

I have since been on ZFS and from my experience on ZFS I have a suggestion for how to do your RAID5 arrays to get both resiliency and speed.

On ZFS I am using Nested RAIDZ aka Nested RAID5. I can have 1 drive fail for each vdev giving me the same as RAID6.

Nested RAID5 would also give you the R/W performance. It is either a Mirror of RAID5 or Stripe of RAID5.

nx2l · April 24, 2021, 11:52pm

That’s effectively raid 50… Not raid 6

LinuxMaster9 · April 25, 2021, 12:35am

RAID 6 is 2 drive redundancy. I am working with Mirrored RaidZ1 arrays:

nx2l · April 25, 2021, 2:46pm

where does it say those are mirrored?

I dont think you can mirror raidz vdevs…

LinuxMaster9 · April 25, 2021, 5:06pm

It has been awhile since I set it up. I think I striped the 2 vdevs in the pool.

But from some lite reading, yes you can mirror 2 RAIDZ vdevs in the same pool.

Using a RAIDz calc you can play with the numbers a bit and see. 2 groups, 4x4TB drives each. Striped groups of RAIDz1 disks.

Additional disks can be added similarly to a RAID-Z configuration. The following example shows how to convert a storage pool with one RAID-Z device that contains three disks to a storage pool with two RAID-Z devices that contains three disks each.

The total available space with striped RAIDz1 vdevs in the same pool matches my available space.

Also, that image is just from ZWatcher. It isnt super detailed. It just gives me the basic layout and if there are errors

bambinone · October 13, 2022, 1:54am

Try lvchange --syncaction check <LV>_corig (or repair instead of check).

This has been very helpful, thank you. I’m pretty well-versed in LVM but I’ve never done this before, and it seems like there are no other guides that cover the case when you want to mirror or stripe your cache drives. I’ve got a stripe of NVMe drives writeback-caching a mirror of SATA drives and it works great!

REDACTED

If your cache data and metadata are on the same device(s), you can skip these two steps. When you run the subsequent lvconvert --type cache --cachepool command exactly as-is it will automatically reserve 0.1% of the cache LV for metadata.

The extra steps are useful if you have e.g. a larger–slower cache drive and a separate smaller–faster cache drive and want to put cached blocks on the former and metadata on the latter.

EDIT: Oops! I am quite mistaken. It will automatically reserve 0.1% of the cache LV for metadata, but it’ll put it on the first available extents in your volume group. If you want it striped or mirrored and/or stored on another device these extra steps are indeed required.

No, you can’t. Sorry. You can’t put a vdev in a vdev, and if you have more than one vdev they are always striped.

LinuxMaster9 · October 13, 2022, 9:32pm

That was quite some time ago. I now use ZFS.

maxa · March 22, 2023, 9:33pm

Thanks for putting this guide together, very helpful. I’m curious about your experience with ZFS. How is read/write performance in comparison to your raid6 setup?