We are a bunch of volunteering software engineers, budding network engineers, and community members running a non-profit community network/ISP. We’re a 501C3 and all doing this part time.
We’re getting a substantial funding grant and need to buy equipment, so let’s go shopping! Let’s say $50k. I don’t think any of us have a real handle on what hardware to select.
I’m speccing out equipment for a non-profit ISP. Our sites are currently connected by VPN, but we have our own AS and datacenter presence in a major carrier hotel.
We need to run our AS across two regional POPs, each with 1x10Gb transit and 1x10Gb circuit with the local internet exchange. We have ~6U per site, power is flexible.
Our workload is going to be BGP peering for our low traffic AS that’s going to receive site traffic over Wireguard from ~5 sites with 10-100 internet users sharing a 1Gb symmetric residential fiber connection. We’ll also want to run some VMs for our ticketing system, education VMs for students, and the sorts of workloads we’d take off of AWS/Azure.
Notionally we were going to put a juniper or mikrotik switch in to receive our circuits and feed that into 2-3 1U servers running our services. From there I’m lost in all the Xeon SKUs and what’s good for what. Core count? clock speed? ram?
What should we buy? What questions should we be asking?
First question is budget
You said 50k, is that for all hardware in both sites? Does that include connectivity costs?
Second question is, out of your budget how much money is left for the 'everything else’s category
Third question, do you want a main and backup site?
Fourth question, how will you handle backups
Fifth question, do you need the VMS to be highly available
6u is tight for routers(you want a pair per site, right?) Switching and servers, without considering storage and backup …
With these questions answered you should now have a budget and a number of servers, and maybe a priority between no second site or more budget (I’d suggest not to go without backups)… and that will give you the rough budget per server you can afford, that you can use to drastically reduce your choice of hardware.
As for hardware, epyc or even ampere might be an option for your use case …
Yes, both sites. It includes connectivity costs for the second site. (We’re already in the main carrier hotel with peering and transit and some hardware now)
$50k would be for both sites, 4-6 servers and 2 switches plus cables. Budget is flexible. If it came to $100k or $20k that’d be fine. It’s really about picking hardware that’d do our job plus a little room for growth.
I think we see the second site as a backup for the first. It’s more about being redundant rather than being in two different places geographically.
I’m not really sure, we’re basically backing up between sites, and we are mostly backing up config. I think if we did heavier data backups we’d use Azure’s S3-like service.
I am not sure. I don’t think it’s super critical, but if we did I think we’d need three servers per site for quorum of some kind (just thinking in terms of what kubernetes tends to suggest)
In our current installation we have a juniper switch and two 1U whitebox router/servers that are about half depth. I think we could adequately do storage and backup within 10x2.5" sleds per U of machine, as we don’t log much and don’t really have much data to speak of.
Epyc could be good, I just haven’t explored them much as I generally associate Intel with BSD and BSD and intel with networking friendliness.
That is ok for the switches/ routers but if you are running VMS with any kind of data you want to keep, then you need backups.
Usually backups are off the place where you run them
So, for each site:
Two switches
Two routers
Two-1u servers for accessory VMS , not clustered
One 1u server for backups
?
For the routers, if you want 10gbit routing and firewalling, and especially for wire guard at > 1gbps, you’ll want something with a fast single core clock and 10Gbit connectivity
For the VM servers, given your load, anything 1U with a single socket and 128GB ram will do.
Any 1U system will have room for at least 2-4x NVMEs that wil hold your small set of VMs, for the backup system something with some hihg capacity drives will do
As for what models, again, set a budget, make a list of minimum performance / featureset for each component (switches/routers/VMs) and then choose hardware that meets the minimum specs and see if you stay in budget, then add speed/memory/storage until you meet your budget …
It will make the process way quicker …
