ZFS High Availability in 2026 - Full How-to and helper scripts

[wendell]

ZFS-HA: Active/Standby File Server on a Shared SAS Disk Shelf (open source)

ZFS High Availability in 2026

huge thanks to MICROCENTER for sponsoring this project!

Micro Center is THE AI Destination: https://micro.center/c07d51
Sign up for a FREE 128 gig flash drive at Micro Center Austin: https://micro.center/e1ae53
Shop Micro Center Ubiquiti Products: https://micro.center/60a478
Visit Micro Center News: https://micro.center/be3f27

TL;DR: Two commodity hosts, one SAS disk shelf, one ZFS pool that is
always imported on exactly one host at a time. NFS (and SMB!) failover via the ZFS sharenfs property. Pacemaker/corosync two-node cluster. Everything is open source and documented \Uffffffff repo with full how-to, validation record, and test harness: [TODO GITHUB]

What’s this running on?

For this video, in addition to the disk shelf, the two computers in question are just what I had for testing: An AM4 system and a Zimaboard system based on a quadcore Intel Atom. Both of these have LSI 9006-8e controllers in them, attached to the disk shelf.

---

The problem this solves

You want HA storage on the cheap: two used servers, one used disk shelf
(eBay NetApp DS460C, HP D2600, Dell MD1200 all work), ZFS for data
integrity, and automatic failover when a host dies. Commercial HA-ZFS
products charge $25-30k for this; the parts cost under $4k.

The catch depends on your disks:

• SATA disks behind a SAS expander can only be owned by one host at a
  time (SATA Tunneling Protocol affiliation). The shelf enforces it, which
  is actually good (hardware split-brain protection), but failover
  requires actively releasing the affiliation.
• SAS disks (or SATA behind good interposers) have no such exclusivity:
  both hosts can always issue I/O. You must add your own fencing.

The repo solves both paths.

What the repo provides

• The handoff primitive: SMP PHY CONTROL operations through the
  expander to release/acquire disk affiliations. Validated end-to-end with
  data integrity checks (5GB dataset, sha256, across multiple failovers).
• An OCF resource agent (ocf:zfsha:Pool) that does the full sequence:
  rescan ? verify exclusivity evidence ? import on start; unmount NFS ?
  export ? quiesce ? release on stop.
• Pluggable handoff strategies (strategies.d/): stp (SATA
  affiliation dance), sas (SCSI persistent reservations), null
  (MMP-only with fencing), auto (probes the disk and picks).
• SCSI persistent reservation fencing (zfs-ha-pr): per-host stable
  keys, register/reserve/release/preempt. The disk itself enforces single
  ownership: every write from a non-owner gets RESERVATION CONFLICT at
  drive firmware level, independent of both OSes. Validated on a
  dual-controller shelf.
• Controlled cutover protocol (zfs-ha-cutover): for disks with no
  transport exclusivity (plain SATA, no interposers). Ordered
  release/acquire with settle delays, preflight checks on both sides,
  refuses to run if the peer is unreachable or still holds the pool.
  Three consecutive full round-trips verified with data checks.
• Boot-time device quiesce: a udev rule deletes the HA pool’s devices
  the instant they appear at boot, so a booting host never probes the
  peer’s disks (no I/O error spam, no racy re-affiliation).
• NFS failover via sharenfs: the ZFS property carries the NFS export
  with the pool import. No separate NFS resource agents. Measured cutover:
  zero gaps > 2s for a 1-record/second NFS client through the floating VIP.
• A shelf health checker (shelf-health.py): SES-based, reports PSU/
  fans/temperature/voltage/slots/IOM status, console + JSON output for
  monitoring tools.
• A test harness: six cutover scenarios (clean move, crash, power loss,
  network partition, stuck export, simultaneous boot) with a durability-
  verifying write client.

The tricky parts (learned the hard way)

1. Stale affiliations after a crash. If the owning host dies, its
   affiliation persists. The survivor must hard-reset the disk’s phy
   (SMP PHY CONTROL op=2) to release it. A simple clear (op=6) is refused
   while the dead host’s phys are still electrically up.

2. Boot-time probing. A host that boots while the peer owns the disks
   gets I/O errors and the kernel drops the disks from discovery. The fix:
   delete the devices from the OS before anything probes them (udev rule),
   and recover with a host-level reset when promoted.

3. MMP (multihost) protection. ZFS’s multihost=on detects ambiguous
   ownership and suspends the pool. Great safety net… but a suspended
   import can deadlock both hosts. The handoff procedure sequences around
   this: export ? quiesce? release? wait for disks to settle? import.

4. ESTALE vs connection drop. The VIP must tear down before the pool
   stops, or NFS clients get ESTALE errors instead of a clean connection
   drop that hard mounts retry silently.

What’s validated

On a NetApp DS460C (IOM12) with SATA disks behind interposers and LSI
SAS3008 HBAs, Rocky Linux 10.2, OpenZFS 2.2.11, pacemaker 3.0.1 /
corosync 3.1.10:

• Clean handoff A?B?A with byte-identical data (sha256)
• Crash recovery (wedged driver ? phy hard reset ? takeover)
• Power-loss recovery path (nexus-loss detection)
• NFS cutover with zero measurable client gaps
• SCSI persistent reservation fencing: peer writes blocked at drive
  firmware while owner holds the reservation
• Controlled cutover: 3 consecutive full round-trips of a 4x3TB raidz1,
  data markers intact in both directions, ~30-40s per direction
• Dual-ring corosync (LAN + private subnet), two-node quorum
• Pacemaker resource failover with VIP colocation
• Boot-time quiesce (udev rule) on both hosts
• 5GB dataset integrity across all scenarios

Full validation record with commands and outputs: docs/VALIDATION.md in
the repo. Field notes (failure modes that cost the most time): docs/ HOWTO.md

Hardware notes (all three paths validated)

• SATA disks, native (affiliation enforced by the shelf): the STP
  handoff machinery solves it completely.
• SAS disks / SATA behind good interposers: no transport exclusivity
  use SCSI persistent reservation fencing (zfs-ha-pr, fencing=pr).
  Verified: the peer’s writes get RESERVATION CONFLICT while the owner
  holds the reservation; reads still pass (type 5 WE-R).
• Plain SATA without interposers (no transport release at all): use
  the controlled cutover protocol (zfs-ha-cutover) for planned
  maintenance when both hosts are healthy. Not an emergency-failover tool.
• Interposer quality matters enormously. Bad interposers mask the
  drive serial, fabricate SAS addresses, and collapse throughput under
  load (150 MB/s ? under 2 MB/s, constant “Power-on or device reset”
  storms). Good ones pass through the real identity and run at full speed.
  Fast diagnostic: check sas_address OUI and serial visibility.
• Cabling: recommend one host per controller. Dual-cabling both hosts
  to both controllers works but forces multipath (each disk appears twice
  per host a same-disk-twice corruption hazard if you get it wrong).
• Compatible shelves: anything with SAS expanders and SES NetApp DS460C,
  HP D2600/D2700/D3600, Dell MD1200, DataON, etc.

What’s NOT solved

• Automatic failover is not wired to the cluster yet. The cutover
  protocol is proven (3 verified round-trips) but is currently a manual
  command, not a pacemaker resource. The OCF agent exists; wiring it into
  a running pacemaker/corosync pair is the remaining step.
• Emergency failover with no transport release (plain SATA, owner
  dead) has no clean answer without PR support there is nothing to
  preempt. Fence the dead host, then rescan and import; MMP is the
  corruption backstop.
• SBD requires a shared-block-capable disk (SAS, not SATA-tunneled);
  the mailbox heartbeat design is stubbed but not yet deployed.
• Marginal shelves eat disks. Four disks dropped off the reference
  shelf during the build (link negotiates, device never enumerates) and
  none were recoverable from the host side all needed physical reseats.
  Budget for this when buying used hardware.

Getting started

The repo has a complete deployment guide (docs/HOWTO.md), the canonical
handoff algorithm (docs/HANDBOFF.md), and a test harness
(tests/). Start with the HOWTO: it’s shelf-agnostic and covers
everything from package installs to the pacemaker resource definitions.

And remember mostly the pacemaker, corosync, HA stuff is what RedHat was designed to support. We really aren’t doing anything super special here.

Repo: [TODO]

GitHub: GitHub - wendelltron/zfs-ha: ZFS HA notes and scripts (homelab type context) · GitHub

Questions/corrections welcome – particularly interested in hearing from
anyone who’s run SAS disks (native or interposer) on a shared shelf like we’re doing here. I was sort of disappointed sata-native (no interposer) was so flaky (Proxmox likes to email me about 1-2 SATA command errors a week without the interposers. Harmless, and this only happens under extreme load. One or two transport errors a week is.. tolerable, but annoying. An interposer seems to fix this kind of failure.)

Oh, did it increase by one? did it now? Okay…

These are pretty cheap on ebay:

DDNS Mellanox + SAS Combo Card Bootstrap Instructions (Linux)

DDN Dual-SAS2308 Combo Card: Host Booting the SAS Side [20261026_221334] [20261026_221303] Card: DDN 04-00232-614 REV 2 (PLX PEX 8732 + 2x LSI SAS2308 + Mellanox ConnectX-3 VPI) Status: WORKING, Both controllers host boot at system startup, disks enumerate, and the shelf is visible. See Section 8. They are not usable for OS volume/boot drive…

combo SAS + IB mellanox/infiniband (10/sometimes 40 gbe) pretty cheap on ebay

[Dutch_Master]

FP video here: Floatplane

[Michael_Harbuck]

   +-------------------------------------------------------------+

   |             HOST TARGETCLI MACHINE (FC Target)               |
   |  [Port 1]        [Port 2]         [Port 3]        [Port 4]  |
   +-----+--------------+----------------+---------------+-------+

         |              |                |               |
         | Fabric A     | Fabric B       | Fabric A      | Fabric B
         |              |                |               |
   +-----+--------------+----+     +------+--------------+-----+

   |   FC SWITCH A (300B)    |     |    FC SWITCH B (300B)     |
   |      [2x Ports In]      |     |       [2x Ports In]       |
   +------------+------------+     +-------------+-------------+

                |                                |
     ISOLATED   |                                |   ISOLATED
     FABRIC A   |                                |   FABRIC B
     LINKS      |                                |   LINKS

                |                                |
   +------------+------------+     +-------------+-------------+

   |   CLIENTS 1 - 5         |     |   CLIENTS 6 - 10          |
   |   (Single-Port HBAs)    |     |   (Single-Port HBAs)      |
   |   [ 1 Port per Host ]   |     |   [ 1 Port per Host ]     |
   +------------+------------+     +-------------+-------------+

                |                                |
                |      +------------------+      |
                +------+  CLIENTS 11 - 13 |------+

                       |  (Dual-Port HBAs)|
                       |  [Port 1] [Port 2|
                       +------------------+

I have a 24disk HPE machine with all SSD ZFS 1x(3x8) array. no hot spares on a 4xport FC card ZVOL mutilpath to two 300B FC switches. Even with a 40gb data backend I like the FC stuff. separated 1gb LAN for OCFS2 so all machines access the ZVOL at the same time with a lock. I tried both ZFS and mdadm RAID. ZFS runs full speed. mdadm not so much. even with CPU offload. The backplane will saturate quick though… still works well. its neat to use an NVME to pcie slot adapter and run a FC/HBA line to a Pi. Each client needs a multipath.conf, and ocfs2/cluster.conf but fun. host runs targetcli. Emulex cards are auto detect for target mode. qlogic see the qlini_mode. 8gb FC stuff is CHEAP !!!

[thr3e]

So this might have been covered and i missed it, but if data is actively being written like say the NFS client is a proxmox host with VM virtual disks on the zfs pool, would yanking the power on the active node result in data loss, corruption, or a need to scrub those virtual disks post-switchover? Is that a scenario you would be willing to test and see what would happen? I don’t have any shelves to test this with currently.

I was actually looking into this as I have come into a number of pLX 88096 pcie switches I plan to implement for NVME storage upgrades in the future, and was salivating over the non-transparent-bridging support combined with controller to controller DMA for heartbeat and shared state mapping. This would remove the sata/scsi/sas from the equation entirely and let me dual attach JBOF’s to multiple pcie switches connected to multiple hosts.

[Michael_Harbuck]

My next project is a Quanta a 12xNVME all flash 1U server with NVMEoFC. I will need to update to 16gb HBA/Switches. I would be glad to share info on Fibre Channel. It is not as outdated as it may seem. 128Gb is already roadmapped.

[wendell]

if the nfs honors write-out, it works. On vmware nfs this didn’t result in corruption in the “good” sas path and the host would hitch for a sec on writes. the nfs stack on vmware would retry the write “because nfs”