Hey, building a home NAS with no ECC and focus on Price / TB. All important data gets backed to offsite. Is ZFS still preferred over ext4 or whatever is the standard for the distro these days ? (Ubuntu server most likely) I will have a mix of many old and new drives. Remember concatenating disks into volumes when I was younger and might want to do that here too.
No real need for online redundancy, got the offsite back up for that.
Thanks
Well ZFS without ECC still has a whole bunch of advantages over other filesystems so I'd say go with it.
But as for redundancy, yeah set that up. Particularly when mixing a bunch of old and new disks. Particularly if you're even considering setting up ZFS, so you should consider at least some level of ZFS RAID.
Of course 
It depends on how quickly you need your data in case a drive fails. If you are fine with having to wait for the (whole) off-site backup to be transferred back, then I guess you can ignore redundancy. With ZFS you might however be able to squeeze some more life out of your drives if you use them in a "RAID"-like configuration, since ZFS will (silently) correct small errors (maybe just 1 bit), that might happen way before the drive is really unusable. These kind of errors would otherwise be considered a drive failure and you would need to replace the disk way earlier.
Other file systems don't do this kind of integrity check at all and will happily forward false data to the operating system that might cause much bigger problems. ZFS in a way guarantees you that it will only forward correct data or no data at all. For this guarantee, ZFS doesn't trust the drives and only trusts the CPU and RAM, which is why ECC is recommended. If the risk of a bit error in RAM is negligible to you, then you don't need ECC RAM.
Either way, @catsay is right in that ZFS has many more tricks that makes it the superior file system. snapshotting, on-the-fly compression and smart caching to just name a few.
Thx both. The bit corrections I guess can only happen if you sacrifice a third of the space up front right ? (RaidZ?) The server (old pc) I'll use doesn't support ECC so not sure RaidZ is the best option then. Looking for unmirrored disk groups. Snapshots sounds very useful though, compression and caching also. I'll check that out.
The time to restore is not important for this server, I'll use Duplicati to put encrypted backups of the critical data in one or two more locations offsite.
Yes....go for it.
I've been running a media server on FreeNas using ZFS for over two years with no ECC memory, keep in mind that this server has data added and removed regularly (lots of reads and writes), I've even had a drive fail and the pool carried on in a diminished state until I got around to replace the drive. This happen of course since I use RAID Z2 array which I would highly recommend, redundancy might not be important to you since you have off site backup but the virtues of redundancy in your pool go a lot further than just having your data safe, it also as I said above will keep the pool operational till you replace a failed drive. (well unless you loose multi able drives at the same time...lol)
Having a nas and a backup place is awesome. No ECC is a downer but not an end game. More than likely Your data will be fine on ZFS and its checksums and well without ECC there is a chance shit could go wrong in ram but its rare.
Consider windows 10 .... No file system checksum... No ECC and it plays games all day fine.
Your 2 steps ahead of a windows gamer 
Couldn't agree more. This is one of the most hotly contested topics concerning FreeNas or ZFS. While anecdotal, I ran my FreeNas box for two years without ECC. It was fine. I have since upgraded my hardware and added ECC to the mix.
If you don't have ECC, you will most likely be okay. No guarantees though. I guess their are no guarantees with ECC either, just better odds!
Just coming in here to add to the echo chamber. ZFS is still awesome. I use it on external drives that go from place to place. ZFS has a feature called "copies" for situations where you've only got 1 drive in your pool. Though it's not particularly great, it's potentially better than nothing, depending on how important your data is.
Also, ZFS's ability to get your data off site is fscking amazing. I will never look back at rsync.
Guess you mean ZFS replication using zfs send? I planned to use Duplicati since it can encrypt and upload to FTP in one go. Can zfs do that?
How much depends on your RAIDZ (or mirroring) configuration, but you will need to sacrifice some disk space for redundancy of course.
First you need to understand the difference between old-school hardware RAID and "RAID" in ZFS. In ZFS all the disks are managed in software, so if you have a RAID card, you need to run it in JBOD mode which means that all the drives are presented to the OS as is, so as if you plugged them into your motherboard's SATA ports for example.
Next you need to know that in ZFS your whole storage is managed in pools which consist of so called "vdevs". Each vdev has its own redundancy and you are free to choose a different method for each vdev if you want. But if one vdev fails, the whole pool fails, so choose wisely (Basically you stripe all the vdevs like in a RAID0, however you can also just have one vdev if you want.)
Now while you could pick simple mirrored vdevs (comparable to RAID1), you will lose a lot of storage since you basically cut the available disk space in half. This is great for performance and the fastest method in terms of recovering from a drive failure, but not very economical. This is where RAIDZ comes in. Usually the number of disks that can fail at once (that is before you replaced it and the data is written back) is denoted by a number after the "Z". So RAIDZ2 will still be fine, even if 2 disks fail at once. (Technically you could also just have one drive per vdev and get the whole capacity of all drives and the maximum performance but of course no redundancy).
Since drive failures happen rather rarely, you are tempted to think that RAIDZ1 will be all you ever need, but beware. The issue is that when you are rebuilding a replaced drive, you put a lot more stress on the remaining drives which might have a similar age or even come from the same batch of drives. This puts the odds quite high, that another drive will fail. This is why the FreeNAS Team recommends that you use at least RAIDZ2 and get your drives from different vendors and maybe even different drive manufacturers, per vdev that is.
The recommended configuration for ZFS vdevs in RAIDZ is a power of 2 (so 2, 4, 8, 16, etc.) number of disks plus whatever number of disks you want for redundancy. For RAIDZ2 that would be 4+2 = 6, 8+2 = 10 and so on. I am personally only running a small pool of 2+1 drives in RAIDZ1, that I am only using for backups, so I am not too worried right now, but when I upgrade, I will go for RAIDZ2 for sure.
I am not sure about FTP but it works over SSH. You basically pipe zfs send into ssh and run the zfs receive command there. So something like: zfs send mypool/mydataset | ssh [email protected] zfs receive -vudF mypool.
The -v option will print information about the size of the stream and the time required to perform the receive operation. The -u option prevents the file system associated with the received data stream (mypool/mydataset in this case) from being mounted. This was desirable as I’m using backup to simply store the mydataset snaphots offsite. I don’t need to mount them on that machine. The -d option is used so that all but the pool name (mypool) of the sent snapshot is appended to mypool on backup. Finally, the -F option is useful for destroying snapshots on backup that do not exist on server.
Source: https://www.iceflatline.com/2015/07/using-zfs-replication-features-in-freebsd-to-improve-my-offsite-backups/
Obviously SSH does the encrypting and authentication, so you don't need to worry about that.
Sorry for the long post 
I wanted to make sure you know all the important details about ZFS, so you can make the right choice in how you configure your drives. Also it might have been because of my passion for this amazing file system. 
comfreak got here first, so a little echoing of that answer. Technically ZFS doesn't encrypt and send in the same command, but it's expected that you send ZFS snapshots over ssh, so...sorta yeah?
The other awesome bit is that you don't have to pour over your whole dataset every time you want to send it off site. Not sure how Duplicati works, but short of it actually integrating with your file system, I can't imagine it being any better than rsync, and rsync is terrible compared to ZFS replication. With ZFS replication, you just send the changes to your data since your last snapshot over the wire. Suddenly getting your, let's say for example, 8TB of carefully collected useless junk off of your server, and on to off-site storage can realistically be done on an hourly basis.
People keep saying that there are storage products out there that can compete with ZFS, but I haven't seen it yet.
ZFS replications where only blocks on the disk that have changed are copied is pretty unique and efficient. I have not seen anything that comes close as a hobbyist.
I use BTRFS raid 1 of course just because I have a mixed batch of drives and sizes and it is more flexible with that.
To complement that, all that ZFS send does is to serialize a snapshot, similar to what tar does to a directory. You then simply pipe that stream of data into your ssh session, like you entered the data manually on your keyboard. So yes, the data is not encrypted by ZFS send but by the tunnel that is created by ssh. You can trust this encryption the same like when you open up a ssh session and type a password there
This is the pr0n of sysadmins
While your fellow sysadmins are still waiting for rsync to collect all the files that have changed into a list of files that need to be sent, ZFS send is already pumping the data through the line. 
That's another thing, (to my knowledge) rsync transfers the whole file if just one bit changed, while ZFS send will send just the file system level block that was modified.
Wow, I need to learn more about what ZFS can do. Great information!
ZFS doesn't need ECC anymore than other file systems.
the "dooms" day scenario, i.e it writing bad data to disk, is VERY VERY low, i.e you will win the lottery first.
That is not true, because (except for btrfs) there are no other file systems that checksum your data and it's this checksumming that is the reason why ECC RAM is recommended/required.
ZFS does something no other filesystem you’ll have available to you does: it checksums your data, and it checksums the metadata used by ZFS, and it checksums the checksums. If your data is corrupted in memory before it is written, ZFS will happily write (and checksum) the corrupted data. Additionally, ZFS has no pre-mount consistency checker or tool that can repair filesystem damage. This is very nice when dealing with large storage arrays as a 64TB pool can be mounted in seconds, even after a bad shutdown. However if a non-ECC memory module goes haywire, it can cause irreparable damage to your ZFS pool that can cause complete loss of the storage.
For this reason, I highly recommend the use of ECC RAM with “mission-critical” ZFS. Systems with ECC RAM will correct single bit errors on the fly, and will halt the system before they can do any damage to the array if multiple bit errors are detected. If it’s imperative that your ZFS based system must always be available, ECC RAM is a requirement. If it’s only some level of annoying (slightly, moderately…) that you need to restore your ZFS system from backups, non-ECC RAM will fit the bill.
No, its not a requirement, not to mention you have to configure the ram and error checking for it to even take effect, can't just plop ecc into smething and call it a day.
I never said it is a requirement for everyone, it depends on what expectations you have from your storage system.
Of course you need the proper hardware along with ECC RAM and the kernel needs to take care of the memory checking, but that is beside the point.
The main problem with his post is that he neglects other reasons of data in RAM becoming "bad". His experience is based solely on RAM that had a "loose contact" from the factory and if it passes memtest86, it will not break for the next 15 years. He thinks non-ECC RAM is enough because he didn't find any data suggesting that it's dangerous and "nothing ever broke for him" but that is all the statistic he can provide in favor of this opinion.
A guy in the comments mentions a valid point why ECC RAM makes sense if you need your storage server to be as reliable and server failures that require restoring from backups to be as rare as possible. "Lots of bit flips come from atmospheric radiation, which Memtest would not find." He continues with a study from Google that suggests that there are around 5 bit errors per 8GB per hour.
Even if you consider radiation etc. to be a low risk, you need to consider the fact that memtest86 is no guarantee that the RAM is 100% intact. What if you had bad luck and the RAM was working perfectly for the memtest86 runs but after the "burn-in" starts malfunctioning? This might seem rare, but I wouldn't call it impossible. Another thing are power fluctuations that might make their way into the memory modules, as mentioned by the same commenter.
The thing that the author of the main article also neglects was mentioned in the comments too:
Also, you neglect to tell your readers what happens if a bit error is indeed written to disk. ZFS can't fix it, because it trusts what comes out of RAM to be correct. When those blocks with the bit error are read again, the problem could be amplified, and things can go spinning out of control. The fact that ECC protects 1 bit for every 8 bits is significant.
You can argue how bad it is, if your data becomes corrupt when you read it, because you can restore from backups, but what if the previously mentioned case happened and corrupt data was written to disk? You will store corrupt data to disk and always forward that corrupt file to the application and cause potentially bigger trouble there, since ZFS doesn't know that it ever wrote corrupt data. In addition to that you will store backups of corrupt data that will not help you fix the problem, so taking the money saved by running non-ECC RAM to spend on backups will not fix the problem that you wrote corrupt data to disk.
Even if the bit corruption rate is 1 bit per 128GB of RAM per month, if that single bit is not corrected, and makes it to disk, you have corruption. It doesn't matter how you're backing up your data, incremental or not. It doesn't matter if you're running snapshots. It doesn't matter if ZFS supports SHA256 hashing of blocks. If a single corrupted bit leaves RAM, ZFS trusts it, SHA256 hashes it, and stores it on disk. Every time an application thereafter tries to correct it, ZFS will revert it, due to the SHA256 sum on disk. And, if you have a stuck bit in RAM, bit corruption will multiply. Quickly.
On a side note: do you ever run memtest86 after the initial build of your system? Probably not, so a DIMM that is starting to die will not be detected until it is probably already too late. With ECC RAM you can detect that and halt the system before bad things happen.
https://pthree.org/2013/12/10/zfs-administration-appendix-c-why-you-should-use-ecc-ram/
Look the person running windows or linux (ext4) is already swinging in the wind. Making a NAS without ECC memory is 8 steps forward towards securing data. Sure a few steps back with no ECC but still way ahead of the pack and a step towards data being secure.
Over time sure you could buy / move to a system with ECC ran and a live offsite backup.
Cant expect people to go from NTFS or EXT4 to best practice file systems backup in one step.
Thanks a lot comfreak for the long response, super useful. To start I'll most likely stay away from striping for the older mixed drives. Have used LVM ages ago, might do that again. I do have some newer 2TB ones that I guess can suit for zfs and maybe raidz.
Regarding replication that sounds pretty neat. Duplicati does however AES encrypt while backing up over secure connection (also incrementally). Since I don't trust the other side I need the actual data backup to be encrypted, not just the connection.