TL;DR, need to build a fast storage server – for around $10/20k
I’m the new sysadmin for a large-ish research compute cluster; 40+ Nodes. This cluster has no storage other than a spare 500gb ssd on the headnode (Long Story). I have the funding to put in place a network storage solution.
I know my way around hardware, networking, linux … but have never actually built/configured a network server or ever touched ZFS before. So I just want to sound my ideas out here to see if I’m being an idiot. Input appreciated. I have the funding to actually buy this thing and I’ll post pictures, so I guess treat this as “What would be the fastest NAS you can think to build thread”
- Saturate a 40gbps uplink for burst file read. No more than around 100gb. Reading data greater than this can be slower.
- General use case is someone starts a batch job on the cluster and all 40+ nodes need to copy files (50gb) to their local storage as fast as possible, then over time will each write data back.
- Have around 5Tb total storage.
- Some Redundancy. Data is not mission critical but I’ll get all sorts of annoyed people if I loose data.
- Max Budget of $20k, ideally more like $13k
- Current infrastructure is 40gpbs, may move to infiniband later.
- 2U would be best. Can accommodate larger but I’ll have to move things around and it’s loud and dusty in there and servers are heavy
- I’m in the UK btw
- Supermicro motherboard based, because I’m a SM fanboy and all the other kit is SM, so I can remote manage things without having to go into the office or even wear pants.
- Get something like this: SM based server with NVMe
- Add a nice intel 40gbps QSFP Nic
- Then get Some fast spinning rust for colder storage, some faster ssds for hotter files, possibly have an optane drive for a cache (need to investigate mobo/cpu compatibility there).
- Am I being dumb with the hardware. is there something more off the shelf / should I just wuss out and call Dell get them to sell me something
- How much Ram/Cpu power is needed. This thing will only ever be doing this. No extra VMs/secret Minecraft servers
- Which OS? (something unix-like obviously)
- ZFS, completely new to it, pointers/guides appreciated
- SAS/SATA/PCIE cards. I’m new to storage tech, If i’m doing ZFS software raid, then I don’t need a RAID card. However, I think I may still need (depending on the chassis/mobo) a non-raid host Bus Adaptor to connect to all the drives. These seem to top out at 12Gb/s. Can I use multiple and split the load? Is this possible? I take it if it is then I’ve gotta keep track of PCIE lanes. I guess if I have enough ram this shouldn’t be an issue.
- In addition to Q5, see this server for example where the hdd backplane is directly connected to the mobo. How can I tell the max bandwidth of that? Do I need to care?
Thanks for reading,