ZFS hangs after removing physical drive from zpool

[oO.o]

To me this sounds like the HBA isn’t giving proper updates to the OS

Other drives connected to the same controller, but which are member of an md raid, disconnect and connect fine without hanging the raid.

Does MD immediately detect the removed drive?

Also, any firmware updates available for your HBA? OEM drivers?

[skullux]

You mean ‘zpool detach’ and ‘zpool attach’? Yep, this works fine. Though, that is a different use case.

No, I meant by using the command eject, as in eject /dev/sdg.

Glad you got it sorted, btw!

Edit: Another way to safely remove a device would be with echo 1 > /sys/block/sdg/device/delete

Ah I see. ‘eject’ did not work: “non hot-pluggable device”.

But ‘echo 1 > /sys/block/sdg/device/delete’ worked fine and did a clean eject that the zpool could handle. Even when performing a ‘zpool scub’ zfs did not hang and worked as expected.

So another sign pointing to the raid controller and driver not properly communicating a hard remove of the drive.

even up to 10 mintues later zpool has no idea about the change of its hardware

To me this sounds like the HBA isn’t giving proper updates to the OS. And…

Tried importing the pool readonly with a live FreeNAS 11. ZFS on FreeNAS actually catches the error, amazing!

Given your experience here, I would suspect it’s how the driver in Linux handles your HBA. Where the driver in FreeBSD is properly handling the hot removal of a device, Linux is not.

Does MD immediately detect the removed drive?

Also, any firmware updates available for your HBA? OEM drivers?

MD behaved like ZFS, as in only noticing the drive was missing when the next io was issued, and not right away when the drive was removed.

So I went onto the firmware update odyssey of HPE servers.

There was actually an available update for the P440ar, from 4.42 to 6.06. Release notes gave no hint about something like hardware failure communication. Performed the update anyways, behavior did not change. ZFS still hangs in the described scenario.

When browsing the abomination called “Hewlett Packard Enterprise Support Center” I also noticed a “CRITICAL” firmware update for the drives we use in the zpool.
Release notes again kinda worthless, but might as well try it.

Though, when in HBA mode firmware updates to drives can only performed in “offline” mode, meaning having to boot the “Service Pack for Proliant” ISO and performing the update from there.

Well, we did that. Instructions again kinda meh, but finally got it working after a few hours. Though, the SPP also updated the firmware of the UEFI, NICs and iLo and some other crap.

Guess what, now the ZFS hot removal works as expected, no hanging zfs module, even when running ‘zfs scrub’ directly after!!!

Sadly, now I cant exactly pinpoint which of the firmware updates actually fixed it, either UEFI, drives, others or any combination of those. In retrospect it would have been possible to the firmware updates one by one. Luckily, we are going to install a second system with identical hardware configuration later on, so I will have a chance at figuring it out completely.

Still super weird that FreeNAS could seemingly handle the misbehaving firmware in the driver, whereas CentOS could not.

Oh, almost forgot, now 2 of the drives’ fault indicator led is permanently lit amber, meaning “complete drive failure”. The drives work fine, SMART says they are fine, the P440ar says they are fine, the iLo says they are fine, ZFS says they are fine, just the drives them selves think they are dead.

GG HP, going to open a support case now.

EDIT: second installation with identical hardware never had the hanging issue. Why, I have no idea.

exporting and importing the zpool cleared the drives’ fault led. Apparently zfs can directly talk to the drives’ controller and convince it that it is dead.

tl;dr when in doubt, always update all the firmwares.

[oO.o]