I’m running into a very curious issue with my root ZFS pool on my workstation. The layout is four SATA disks in an RAIDZ1 and four M.2 disks in a striped mirror metadata special device. I use special_block_size = recordsize to selectively store my root filesystem datasets on the M.2s. This setup works great.
Recently I decided to add a dataset for ~/.cache, so I won’t waste time backing this trash up to my home server, and inevitably within an hour I start getting metadata checksum errors from the M.2s. The computer then promptly hangs completely. If I reboot, log in as root, and run a scrub (ie, don’t touch my user account and its ~/.cache), everything works fine again and there are no checksum errors. Obviously I’ve stopped mounting the .cache dataset, but it still seems a very curious bug to me. /home is a ZFS dataset stored on the special devices in exactly the same way .cache was, so I see no reason why one setup would work and the other wouldn’t.
I may have wasted a weekend worrying my drives were suddenly dying all at once (even though they’re from different batches) while I was figuring this out.
1 Like
Curiouser and curiouser. If I try to destroy the cache dataset, the same special device checksum errors return. But the same also happens if I try to do a nixos-rebuild. Potentially my /nix dataset is also affected? It would seem that reads don’t trigger any problems, but writes do. I guess I’ll try recreating the pool before I get any new hardware, but maybe it is the hardware anyway. If it were just one M.2 I’d just assume the bifurcation card I’m using can’t handle PCIe4 properly, but zpool status claims it’s all of them.
1 Like
if its reading fine but writing bad data then its likely the cpu or ram causing the write errors.
so first check your ram isnt throwing constant errors.
if your running bios tweaks such as an all core oc. turn it off and try auto boosting.
the ram, test with mem test.
if all is fine then look at your disk config and see if you have any irq conflicts.
2 Likes
Edit: was thinking of when a vdev itself was modified, like when Wendell messed with zfs corruption, not the data inside it. Not relevant here.
@Susanna please excuse me for being a stub, but .cache works fine as part of another array, with special vdev for all of / or /home, but when separated, then ZFS reports checksum errors? And the seperate dataset, is in the same pool, so shares the same special vdev?
I don’t have a solution (apart from not separating it) but am very nosey (please forgive me)
I wonder if one creates a whole separate zpool, without a special device, and moves the .cache to that, if the same errors occur?
I don’t see why it would
I had the same thought, but it’s writing data to another pool just fine, and no errors in memtest (not that this means much, it’s not ECC memory so god knows what it’s actually returning). CPU overclock also is stable for things like Cinebench.
My dataset bubbles/root (and its children) is on the special vdev, the cache dataset is bubbles/root/home/cache.
The pool crashes if I try reading the cache so I can’t try moving it to another pool, but since it’s just a cache I’m just not going to bother.
1 Like
https://docs.oracle.com/cd/E19253-01/819-5461/gbbwl/index.html
have a look here. its the admin doc for zfs so might be of use.