Hi, due to faulty UPS I got power surge on working workstation and unfortunately due to that my SATA RAID10 ssd array got corrupted.
[ 6871.012584] BTRFS info (device dm-7): first mount of filesystem c0e978a2-29c3-4589-b72c-65ee9881c371
[ 6871.012604] BTRFS info (device dm-7): using crc32c (crc32c-intel) checksum algorithm
[ 6871.012610] BTRFS info (device dm-7): using free-space-tree
[ 6871.014355] BTRFS warning (device dm-7): devid 1 uuid f48c0a4e-ae88-46f1-b7a9-044271c6f763 is missing
[ 6871.025313] BTRFS error (device dm-7): parent transid verify failed on logical 7095897686016 mirror 2 wanted 93529 found 93527
[ 6871.025582] BTRFS error (device dm-7): parent transid verify failed on logical 7095897686016 mirror 1 wanted 93529 found 93527
[ 6871.025589] BTRFS warning (device dm-7): couldn't read tree root
[ 6871.029701] BTRFS error (device dm-7): open_ctree failed
I’ve been told on IRC by devs that behavior like this indicates issue with write barriers. Which might be partially true because it’s second time this particular array got trashed, while exactly the same workstation has second, NVME backed btrfs RAID10 which also suffered power surge many times and never failed like this.
I found some threads about dm-crypt being a bit finicky with write barriers support and I do indeed use LUKS beneath btrfs on those drives.
Is there any way to verify whether my stack properly handles write barriers?
your storage stack doesn’t have working write barriers. What you can do is disable write caching on the drives and/or on the controller (if it has write cache) or get hardware that has working barriers.
the error shows that there was a missed write on both devices so there’s a kernel bug or something in the storage stack doesn’t have working barriers/FUA.
SInce it only happens on those particular SATA drives out of all my machines I would rather assume issue with barriers. Is there anything I can do to make my storage more synchronous? This machine has butlload of RAM and I’ve already been bitten by this fact few times in the past because Linux seems to be very happy to cache 250 GB of writes in RAM and then just f*cking dump on kernel panic or whatever, resulting in data loss on half of array…
Transid difference was small and btrfs managed to recover itself from backuproot but still - this time it worked, next time it may not. I want to do something to prevent that from happening again (apart from replacing 8 years old battery in UPS). My storage stack looks like this:
sda 8:0 0 7T 0 disk
└─dcraid3 254:9 0 7T 0 crypt
sdb 8:16 0 7T 0 disk
└─dcraid 254:6 0 7T 0 crypt /dmnt/dcraid
sdc 8:32 0 7T 0 disk
└─dcraid2 254:8 0 7T 0 crypt
sdd 8:48 0 7T 0 disk
└─dcraid1 254:7 0 7T 0 crypt
So it’s LUKS laid on plain bare drives, and then btrfs on top of dcraid[0-3] devices. I’m using onboard SATA controller on WRX80 motherboard.