There are other options but whether they are better or not is arguable.
That's why your system is dying when many people access it.
RAIDZ2 is Parity RAID. Parity RAID requires CPU horsepower to calculate the parity bits needed. So accessing storage intensively means pushing the CPU hard.
I'm going to guess you guys use Windows machines. That means you almost certainly use SMB for the Network Share.
SMB is Single Threaded. Meaning, when you are pushing the NAS hard, it only has a single 2.0-2.4 GHz Atom core to handle SMB traffic and three 2.0-2.4GHz Atom cores to handle Parity calculations.
Faster storage means more CPU overhead due to more parity calculations necessary when you are using the extra speed.
Overall, this is what I would do:
Get a new server and install FreeNAS or a flavor of Linux Server edition and use BtrFS.
Here's an article looking into why ZFS and BtrFS are good: https://arstechnica.com/information-technology/2014/01/bitrot-and-atomic-cows-inside-next-gen-filesystems/
Realize BtrFS doesn't have the same requirements ZFS has if you decide to do that instead. I'd recommend FreeNAS unless you are already familiar with Linux in general just because of ease of use. However, ZFS optimization isn't exactly easy to do.
Either way you choose to go, do not use parity RAID. The benefit of Parity RAID isn't worth the costs imo, especially when what you need is performance over reliability but you still need reliability.
RAID 10 (a mirrored then striped RAID array, or vice versa, doesn't matter usually) is better in my opinion. Let's compare the two.
RAIDZ2 is comparable to RAID 6. RAID 6 is parity RAID with two drives worth of redundancy. You can lose two drives without losing access to the data.
RAID 10 has between 1-2 drives of redundancy in a single mirrored array (you can go bigger, but you probably don't want to). It has between 1 and 2 because it depends which drives die. Let me illustrate:
If 1 drive dies: You lose nothing. Ever.
If you lose Disk 1 or 2 and Disk 3 or 4, your array will live and continue to work.
If you lose Disk 1 and 2, or Disk 3 and 4, your array will be lost.
So you can lose 1 of each half but you can't lose both of one half.
So it has more redundancy than RAID 5 but less than RAID 6, effectively.
You're gonna have backups so losing the array isn't a big deal for data at least. The odds of two drives dying at all is very low and even if it happens, it's possible it won't matter. So reliability is pretty good as well.
The bigger reason to choose RAID 10 is because of performance.
Smart Operating Systems like FreeNAS can use all four drives when reading data. Meaning RAID 10 is effectively RAID 00 for Reading data. For writing, it's still RAID 0. So you are using four HDDs to read and two (effectively) to write in terms of "how much performance am I going to get?"
So it gives adequate (in my opinion) reliability and redundancy while giving a boost to performance.
Do not try and use an SSD for caching on ZFS. I say that because it is effort to get that working right and it isn't intuitive at all. Only much larger storage setups would benefit from it, and ZFS already has a cache in the RAM which is fine for smaller storage servers. Smaller here meaning Less Than 30TB. But even then, you'd have to consider number of files VS the size of the files. It gets pretty complicated and isn't worth the effort when there are easier and better options for improving performance.
It also doesn't use Parity so it doesn't put a load on the CPU as much. This means you can dedicate it to other things like checking data for bit rot. If you switch to another network protocol instead of SMB or update to something that can handle SMB 3.0 which is not single-threaded, then that means it should be less of a bottleneck, if any at all.
The biggest concern with RAID 10 is the storage loss.
This setup halves your available storage. So four 4TB drives gives you 8TB of space. So you'd probably want six 4TB drives for 12TB. This means you are reading from six HDDs and writing to three (effectively).
It's a concern but I think since you specifically need performance, it's worth the cost.
I wouldn't expect to spend more than $3,000 on this for multiple reasons. For one, you're a small business. There are small business solutions for these types of problems that aren't nearly "$5,000" expensive.
My biggest reason is simply "you don't need enterprise hardware for what you wish to accomplish, though it would help." A bigger concern is replacement. What happens when something does die? Do you go used shopping again for that specific part?
That's a level of unreliability I don't think you want in your primary workhorse server. Buying parts that are readily available now would be best, I would think.
Considering the "why?" of why your current setup isn't working, you shouldn't need much more, imo, than to reconfigure and simply get a better CPU.
What I would do in your situation.
Get a Prosumer solution. Buy hardware in that weird product area between consumers and professionals. I'd even suggest just getting a RyZen 7 CPU if ITX boards existed for it. mATX would be fine too since you'd need the storage space for six or more HDDs. AM4 motherboards are looking to support ECC RAM, so they're just shy of server hardware anyway. And as you may already know, RyZen 7 is pretty good at Production specific workloads. Which is what servers do.
i.e. I would just get this in this setup and call it good if I were you:
- RyZen 1700 (To be honest, I feel like for a NAS only, this CPU is overkill, but imo I'd rather have more than less and it's Price:Performance is great).
- Some mATX board (because servers don't have to be big and there's little reason to go bigger than this).
- Greater than or equal to 32GB of ECC RAM
- Whatever mATX case fits everything you need (6 or more 3.5" HDDs)
- Install FreeNAS 10.
- Configure with ZFS into a RAID 10 setup (basically create a stripe of 3 drives, then mirror that to the other 3 drives)
The cool thing about RAID 10 in the above configuration is it is easy to expand redundancy and thereby read performance
Just mirror three more drives to have more redundancy. Then you'd have a RAID 10 with three drives striped and two sets of three drives mirroring those. Then 3 drives have to die before any data loss occurs, but you can handle up to 6 drive failures depending on which ones die. This improves performance too because then you're reading from more drives (though I feel like that won't scale well).
To improve writes, or to increase overall storage, you'd have to tear down the array and rebuild it. i.e. undo the RAID then stripe 4 drives instead of 3.
Here's a thread on expanding a RAID 10 in ZFS: https://forums.freenas.org/index.php?threads/need-guide-to-set-up-raid-10-in-freenas-8-3.10652/
Here's a guide for things to not do on FreeNAS for setting things up: http://www.freenas.org/blog/freenas-worst-practices/
That's a RAID 10. If you want to add redundancy/read performance, you make more Columns. If you want more storage/write performance, add more rows.
Columns = mirrors. Rows = stripes.