Hi All. I have a question, which might be a stupid question, but I do not know how to properly search for the answer.
I am from South Africa and I want to host some servers in a data center. These are Supermicro servers and I am looking at buying these switches as well: Supermicro | Products | Networking | SSE-C3632S/SR
I will be registering for an ASN and renting my own IP address block. The link provided by the datacenter can run at “any” speed, depending on my hardware. I was hoping to get 100Gbps.
So, I guess the question is: Would I still need a router between my switches and the cross connect of the data center?
It depends on how big your IP block is and how many servers you have. If you have more servers than IPs, then you’ll have to NAT. Also, if you’re going to be doing any firewalling on the network-side, it might make sense to have a central router as well (Although it is possible to do transparent firewalling on all incoming network connections without a router)
If you’re bringing your own IP blocks you’ll need some way of announcing that IP block to the world …
… if we’re talking about ASNs, you’ll probably want to have something speaking BGP and announcing that IP block to other things in that datacenter, (or perhaps outside the datacenter building itself) who you happen to be peering with.
… that piece of software that announces routes can run on a regular server (ideally more than one), or it can run on a traditional separate router appliance … which are basically just glorified network switches.
That said, and judging by your question…
… are you sure you need an ASN? How many machines or peers or IPs are you planning?
I’m asking because I’m imagining most people would upgrade their way towards needing an ASN gradually, they’d typically already have all the stuff in place before considering some kind of multi-homed setup, and at that point they’d already have a network admin telling them that’s what they need.
(e.g. typically you start by having a few racks of your own in a datacenter and datacenter where you’re renting space can introduce you to folks providing transit and can arrange some IP subnets for you… if you have 2 datacenters where you have gear and transit and want to load balance, you start thinking about running an AS, … or if you have a single facility and want to buy internet transit from multiple people offering it, … or if you want to peer with Google/Amazon/Microsoft directly because you have a mixed workload).
So the idea is, I will have a couple of servers, running K8s. I will be hosting websites, so each node running e.g: nginx, will have a “public IP address” pointing to them. So the amount of servers might be low, but there will be a couple of nodes.
I will be visiting the datacenter in the new year, but from the quote they provided, I just assumed that I will connect my switches (configured with routing policies and BGP) to their cross connect links and boom, I will have internet access. It seems I am wrong in this regard and will need to research more.
Yes, getting a network engineer would be the right way to go, but I have the need to learn these things myself and this, I feel, is a great oppertunity.
when you say “a couple”, you mean 2-3 machines … or 20-30.
The particular switch “SSE-C3632SR” has an Intel Atom inside for the control plane, and runs Cumulus… which is basically a mangled Debian, … but it ships with Quagga … which is this wonderfully ancient and somewhat well known router toolkit software suite… which just so happens to supports BGP (thus making the switch you’ve chosen support BGP).
So, you could use it if you needed to.
Since you mentioned “opportunity”…
If you want to learn BGP basics, you can gloss over the CCNA/CCNP routing and switching materials, … official/unofficial, from wherever. You can try playing with cisco ios in GNS3 or try bgp in packet tracer…
In addition to e.g. cisco ios / junos … you can play with regular old Linux in VMs … e.g. install ubuntu and apt install quagga and go to town that way.
Once you’ve figure out basics at slow speeds,… if you want to process web traffic at 100Gbps+ rates, also please look at Loadbalancing Demstified talk, Google Maglev paper, and Facebook Katran
Thank you for the information. That is very helpful.
The only reason why I haven’t played with cisco ios, is because I didn’t want to confuse myself with the command, but I guess it’s more about how it works, thant he command you use to make it work.
I know, but the company is already in the process of buying the hardware, so no use spending more on RPIs and smaller routers, when I already have this equipment.
I think you should heed @oO.o 's warning and hire someone to assist with the deployment. On a 100Gbps connection you could rack up $1000’s/day in outbound transfer fees if your servers were to get compromised.
I mean if money is no object, yolo it, but otherwise, yeah I’d hire someone and at least have them train you on it to avoid very costly trial and error.
13 servers is not that much… not worth investing into fancy L4 load balancing IMHO, from a business side I’m estimating it’s not worth hiring an in house network person - you won’t be running constant projects past initial setup (good ones are not cheap anywhere).
You can contract out the setup+network maintenance to an MSP for O($5k) - O(10k) / year probably… (if setup correctly it’s just basic monitoring and maintenance, there isn’t much to do except keep up with updates, and keep procedures in place and up to date). They’d charge you extra for initial setup.
Specifically, in terms of hardware I think you’re likely looking at a second switch for some basic 1+1 redundancy in routing setup.
In this topology each machine gets one link to each switch and switches get a couple of links in between for failover type stuff.
When stable, you can announce half of your IP space through one switch, other half through the other. In case it goes down traffic converges to surviving set of links/ports.
This way you get high throughput within your rack, for VM or container migrations and remote storage type of stuff, and you get a bit of availability for your machines as well.
Also worth keeping in mind is that each of your machines is a router as well for your containers and VMs… you need to get that routing story straight as well.
In terms of other stuff taking up space in your rack:
you’ll most likely need a small cheap VPN firewall/switch appliance for ipmi/OOB management for switches and machines… this is cheap but takes up space. (either a hardware VPN, or a cheap L2 rack switch and a cheapest rack mountable Ubuntu server box for VPN)
-PDU… Various DCs and colos will have redundant power for high availability, but usually they’d require you to get the power from 2 separate rails… which is why your servers have dual power supplies typically. check the details with your DC.
Okay, so 6 months later and I have learned a lot and you guys all helped in that regard.
We now have our own ASN and /23 IP block. To answer the original question, yes I do/did need a router, especially from a known reputable brand.
I got BGP working on the SM switches that we bought, but I am unable to filter routes and have a ticket open with SM and they have created a firmware update for me, but still not working. Apperantly they are still busy with this.
I just want to thank you all again and sorry for bumping this “old” thread.
