I’m extremely new to using ZFS and there are a few thing which I want to do. I will state the reasons down below.
Turn off ARC and all kinds of write cache.
Get a prompt before ZFS repairs a rotten bit of a file. Or if that is not possible, get a log of which file was “repaired”.
Optimize it for an all-SSD pool.
Now that I’ve stated what I want to do, I’ll talk about why I’ve made such “bad” choices. I live in India - where computer prices are ever so sky rocketing and where old quad core servers with mere 8GB ECC RAM cost at-least $400 (yeah it’s cheaper in USA but it ends up being the same cost-wise if I ship the said server in India, so not really worth it). The only reason why I chose ZFS is because of it’s checksumming and repairing features. All I need from a file system is for it to checksum the data being written in real time - which ZFS does, and for the file system to periodically check the whole disk for any kind of bit rot - which ZFS does.
Now, because of high prices of new & old hardware, ECC memory is out of question over here in India. I will be ordering a workstation for myself with Ryzen 9 3900X (yes I know Ryzen has un-official support for ECC memory, but I either want all in or nothing at all - no hanging in between for clear and blunt “I have ECC memory” or “I don’t have ECC memory” in terms of troubleshooting & support) for Deep Learning, I have it equipped with 5x 1TB SSDs with the intention of getting 3.5-4TB of available space in raidz1.
Reasons why I’m going with SSDs instead of HDDs
The storage pool will only be written once and is purely for archival purposes, so exponentially faster scrubs because of SSD’s read/write speeds.
Even if a disk fails, that rebuild speed even of a SATA SSD is a godsend.
As I’m only using SSDs, the write speed impact when turning off ARC won’t be a problem for me. Also as shocking this might sound to the majority of you, the SSDs I’ve chosen are consumer TLC SSDs and I do not intend to use them as a fast editing NAS so there’s that. I’ll only be filling at-most 1.7 TB of data in the 3.5-4TB pool once it is all set up and writes will be at most 10 GB of files, only once a month. I can’t exaggerate this enough. I selected SSDs because I need to spend less time cringing + being nervous when a scrub occurs and in future when I need to replace a disk and resliver.
Talking about my intended setup, please bear with the insanity of my “intended” setup; I will be setting up a Ubuntu server VM and use
zfsutils-linux for the ZFS part of my setup. I will perform the optimizations which I asked for help in the beginning of the post and then setup the 5x passed through SSDs in raidz1. Then I will transfer all the important data I have to the SSD pool. I also have an external 4TB USB HDD and a Raspberry Pi 4B 4GB. I will set that RPi with Ubuntu server as well and use the single HDD in ZFS as well (this time I don’t need any of the above requested optimzations). That Raspberry Pi setup is just for redundancy as you’ll read about it more down below.
Reasons as to why I have opted for such bizarre setup.
Can’t justify ECC memory with my budget so I want to turn off any writes in RAM if possible. As, if I have understood ZFS correctly, ZFS will first write to ARC and then to the disks. So a bit flip in non-ECC memory won’t be detectable at the time writes. Also because I have pretty fast pool write speed considering all of it is just SSDs.
I requested to get a prompt before ZFS repairs/attempts to repair a file is because I have non-ECC memory. I want to make sure if the file is actually corrupt myself. If it is corrupt, I will let ZFS go ahead and if it isn’t, it’s most probably a RAM issue and I will try to see if a reboot helps. Or else, I do have a remote copy of the same file on the Raspberry Pi
If getting a prompt before ZFS repairs a file is not possible, I want to get a log entry regarding which file was “repaired”. I will perform the same verification as I mentioned in the above point myself as I have no problem doing a bit more work in the absence of ECC memory.
This will be my first ZFS build - as it clearly appears so. So any help/recommendations regarding my setup will be extremely well received. :’)