Please excuse the long story but the details are important. I’ve had a rock solid workstation for some years now with a ASROCK B650E PG-ITX, 2x p4510 Intel U2s, 7950x, and 96gb of ECC. It is not a system I could afford to buy again at today’s prices and the resources I’ve available, but there appears to be a ghost in this machine! I shouldn’t have, but about a month ago I did a BIOS update on the motherboard and when I booted back into the computer it seemed like just the storage was behaving very slowly. Low risk I thought – two enterprise drives with flawless smartctl stats which have a ZFS mirror across them. However, the booting taking about 10-20 times as long as normal had me turn it off quickly, downgrade the BIOS, and I was happy to see everything was fast again. So of course I do a zpool scrub, it comes back perfect, and I’m relieved. About 10 minutes later I start noticing CKSUM errors in the ZFS mirror for the first time in the life of this system so I decide to scrub again and this time CKSUM is counting up rapidly in the thousands – equal CKSUM errors across both p4510s in the pool. At this time I note there are still no ECC errors. And the p4510s’ smartctl stats are still flawless and it is obviously impossible that they’re both failing in this manner – so I realize it is possible that the scrub is making it worse as it tries to fix things and that somehow the storage controller (or something on that order) is somehow corrupting all of my data! On this system the storage controllers for both NVMe slots (which have adapters for the U.2 drives) are on the CPU itself. During my career I never had a faulty controller destroy an HDD array – but I knew such things could happen and I thought a CPU-based controller would even be less likely or possible than that!
So within about a minute of watching zpool scrub count up on the CKSUM I freak out, cancel the scrub, and shut down the computer. Then I got busy testing all of the hardware and I could find no issues with any of it. CPU, RAM, SSDs all go through tests flawlessly and have flawless stats – and I’m relieved because I can’t currently afford to replace it - especially out of superstition alone. So I assume all of the damage happened somehow while running that new BIOS that made the SSDs hang – but how can that be when right after the BIOS downgrade I had a full zpool scrub without a single CKSUM error (before the 2nd scrub where the mirror’s CKSUMs counted up in tandem)? Perhaps I lack appreciation for a class of data corruption phenomena that can happen in computers which is entirely transparent to ZFS - or which ZFS would exacerbate if a scrub were issued.
I will bend your ears later regarding ways to recover the corrupted data, which was not entirely backed up. I gave it a couple days of work and have failed so far. The ZFS pool being encrypted does make that more of a challenge. I did immediately ddrescue both entire p4510s in that mirror, so I can try more seriously to recover that some day. Of course mounting each drive’s image and trying to scrub it or scrubbing them together basically gives me the same results with thousands of CKSUM errors – I guess somehow the computer really did write all over those drives.
Now fast forward to about a month later where I’ve been using the same hardware again assuming all of that was caused by the BIOS update. This last month gave me zero notifications of ECC errors, smart errors, or zpool errors… Now this morning I wake up to the power button blinking as if it crashed and it is memory testing, but it never stops. I get it back up with a CMOS reset and I feel that pit at the bottom of my throat: zfsbootmanager says no pools to import. Holy hell, again? The computer was running flawlessly with zero notifications about any ECC, smartctl or zpool errors. But yesterday a flatpak did require a –repair in order to upgrade and then it complained of a lot of checksums not matching! So perhaps corruption was happening suddenly and transparently from ZFS’s PoV again?! Now I’m sufficiently demoralized and disillusioned to turn for help from and in search for experts hopefully a lot more adept in these topics than myself… My attempt at designing my most reliable computer ever has mutated into - at a moment’s notice - the least reliable computer I’ve ever owned.
I need this workstation to keep food on the table as I’m self-employed. Back when I used to work in big tech it wouldn’t have been as much of a deal because it was easy to get replies from vendors and you’d get spare parts at no personal cost to hold you over. And because we were engineers there was a culture where they wouldn’t just throw out the part and give you a new one, but actually debug so everyone can learn about true cause & effect – a prerequisite for designing better systems! I’d love to be able to do that in this case, but at this point I’ve accepted I can’t do it without help.
Other potentially relevant details:
- This is a zfs-on-root Linux install
- kernel 7.2 and ZFS 2.4.3/2.4.4
- StarTech M2E4SFF8643 adapters for NVMe → U2