ZFS vdev_id and Multipath SAS

Hardware Hub Storage
, ,
1

Hi guys

Because I work with NetApp professionally I have a lot of older equipment like the DS4246 and later DS212C/DS224C shelfs… I use MiniSAS HD cables with the IOM12 modules installed.

Over the years I have just been using one IOM12 module and one SAS cable connected from the shelf to a Linux host.

I learned early on that using the /etc/zfs/vdev_id.conf made sense as it was able to number the disks based on the drive slot… so for example, a typical vdev-id.conf file would look something like:

multipath     no
topology      sas_direct
phys_per_port 4
slot bay

enclosure_symlinks      yes
enclosure_symlinks_prefix shelf

channel 01:00.0 0         0a-
channel 01:00.0 1         0b-
channel 01:00.0 2         0c-
channel 01:00.0 3         0d-

alias intssd03 /dev/disk/by-id/scsi-35002538b0236c670
alias intssd04 /dev/disk/by-id/scsi-35002538b0238d8f0

If you create your zpool using the /dev/disk/by-vdev/… you can get a nice zpool like this:

pool: aggr10
state: ONLINE
scan: resilvered 1.76M in 00:00:01 with 0 errors on Sat Sep 5 15:48:41 2026
config:

NAME                   STATE     READ WRITE CKSUM
aggr10                 ONLINE       0     0     0
  draid2:21d:24c:1s-0  ONLINE       0     0     0
    0c-0               ONLINE       0     0     0
    0c-1               ONLINE       0     0     0
    0c-2               ONLINE       0     0     0
    0c-3               ONLINE       0     0     0
    0c-4               ONLINE       0     0     0
    0c-5               ONLINE       0     0     0
    0c-6               ONLINE       0     0     0
    0c-7               ONLINE       0     0     0
    0c-8               ONLINE       0     0     0
    0c-9               ONLINE       0     0     0
    0c-10              ONLINE       0     0     0
    0c-11              ONLINE       0     0     0
    0c-12              ONLINE       0     0     0
    0c-13              ONLINE       0     0     0
    0c-14              ONLINE       0     0     0
    0c-15              ONLINE       0     0     0
    0c-16              ONLINE       0     0     0
    0c-17              ONLINE       0     0     0
    0c-18              ONLINE       0     0     0
    0c-19              ONLINE       0     0     0
    0c-20              ONLINE       0     0     0
    0c-21              ONLINE       0     0     0
    0c-22              ONLINE       0     0     0
    0c-23              ONLINE       0     0     0
special
  mirror-1             ONLINE       0     0     0
    intssd03           ONLINE       0     0     0
    intssd04           ONLINE       0     0     0
spares
  draid2-0-0           AVAIL

When a disk fails, I know which slot it is installed and I don’t have to fiddle with serial numbers etc..
There is even the possibility to turn on the yellow LED on a drive slot..

Anyway the reason I am creating this post, is that I would like to use multipathing.. so installing another IOM12 module, and connect another SAS cable back to the server…

This is all possible, and with SAS drives it makes some sense, and maybe even a bit better performance.

But I need some help, because while it works fine with the multipathing, I need to import the pool using the /dev/mapper/mpath… and that’s the problem :slight_smile: Do I loose the slot mapping when using mpath? Does anyone know that the “multipath” option in the vdev file actually does? And does it make sense here?

The thing is that vdev has the" “slot bay” option which makes the slot pairing possible, and I would like to keep that some how, while adding mpath…
But is it even possible? And has anyone tried it?

If I find the time, I will do a few tests to see if I can get it working :slight_smile:

2

It makes it process multipaths based on their first device. You have to already have multipath setup (either in auto-find or statically). IIRC, it also prevents the child drives of that multipath from being enumerated individually.

IIRC, you also want to give each path the same device channel name, the manpage ‘vdev_id.conf(5)’ gives the following example:


     A multipath configuration.  Note that channel names have multiple definitions - one per physical path: 
           multipath yes 
 
           #       PCI_SLOT HBA PORT  CHANNEL NAME 
           channel 85:00.0  1         A 
           channel 85:00.0  0         B 
           channel 86:00.0  1         A 
           channel 86:00.0  0         B

Also, having vdevs based on position means that autoreplace can automatically “replace” a dead drive with a new one that gets put in the same slot.

3

I had the same issue as you :slight_smile: or worry should I say. I originally had my ZFS cluster on a CSE-847 which I exported the pool. I took those drives and moved it to my DS460C with dual connections and multipathing.

I recorded all the drive sizes and serial numbers (and SAS Address) before the move of creating the pool using the device name itself. After I put them into my new system I went through and mapped all the new device mapping (for multi pathing) to the same zpool command but instead of create you should look at the zfs import as you can give it the -d device for the pool. This rebuilds the pool and uses the mapping. You should only do multipathing imo if you are using SAS drives not worth it for SATA.

cat /etc/multipath.conf
defaults {
    user_friendly_names no
    find_multipaths yes
    path_grouping_policy multibus
}

blacklist {
    devnode "^(ram|raw|loop|fd|md|dm - |sr|scd|st)[0 - 9]*"
    devnode "^hd[a - z]"
}

I used the guide from Ubuntu as a helper to figure out the configurations. Both paths are active and in the same priority group. Try the commands out on some drives you do not care about, and as long as you do not use -f with zfs it will be fine :slight_smile:

Linky Multipath configuration examples - Ubuntu Server documentation