So I'm just goofing off with some hardware.
I'm going to do a video on FreeNAS and setting it up properly. For now I'm experimenting before buying any hardware to round things out.
I expect that the Netgear XS712T and the Asrock X99 10G motherboard will work well as I've already tested them and they work great. I will be using Jumbo Frames (eventually) and link aggregation as I have already tested that and it works great with this combination.
Now I am testing the minimal drive configuration for maximum throughput and redundancy. I suspect, but am not certain, that once I get my hands on some PCI Express NVMe SSDs I will be able to configure a NAS/SAN for absolutely stellar speeds. However, in the past, I've hit limits with SMB and the FreeBSD kernel (8.something) that required extensive tuning.
I think Linus' team is also working on some kind of set of behemoth network servers, but I suspect I'll be able to build a Linux or FreeBSD based box that matches the performance but for a fraction of the cost.
So I thought I'd share what I'm up to here, and where I expect to go.
The name of the game is
1) At least 2 gigabytes/sec sustained read
2) Ideally, 2 gigabytes/sec sustained write
3) Try to approach this speed with silly windows file shares (e.g. cifs -- these speeds are way easier to achieve if you're doing it with iSCSI which I have in the past and it was a cakewalk)
4) Gracefully handle about 4 I/O heavy operations simultaneously
Hardware I'm using or expect to be working with:
Netgear XS712T
Asrock X99/WS Mobo
Xeon E5-1650v3
64gb ECC DDR4
Some additional Intel 10gigabit ethernet adapters
For Drives we have:
4x Samsung Evo 850 (well, actually, I have quite a lot of these on hand, but 4 is the minimum to clear 2 gigabytes/second, and the current config )
8x Older 750gb 7200 rpm Spinning Rust WD Blacks and Seagate 7200.12 HDDs
For the case, pictured here is an old SuperMicro case with 8x 3.5 HDD bays (probably will use a Norco or SuperMicro 20-24x 3.5 bay HDD case later). I have configured it with 8 2.5 hot swap bays using the two internal 5.25 bays.
I think in another chassis I'd probably do 16-18 spindles + 8 SSDs for caching/speed (or 4 NVMe SSDs because they are 2x as fast+). Knowing how to size it is part of what I hope to get out the experiments here.
This is just a test box -- to check and see if the assumptions I have are correct, and what's necessary to thread the needle for having something crazy high performance but also not wasting resource on some kind of insane overkill.
Anyone have a link to the tweet with Linus' groups numbers so far on what they're building? IIRC it's two storage boxes + recycling their old server into a freenas box but freenas sill needs a bit of tuning out of the box to be faster than 2-3 gigabit.
I've gotten to a max of about 7.5 gigabit with 5 minutes of tuning and I haven't even done anything really special yet (and no link aggregation so far).
So for the disk speeds I already have with this really weaksauce configuration, the network is the bottleneck so far. (8xhdd in RaidZ1 pool + 4 SSDs for cache)
