Network services for self-hosted bare metal kubernetes cluster

Hardware Hub Networking
, ,
1

TL;DR : I don’t know how to set up a productive network for my self-hosted kubernetes cluster with limited hardware. I’m thinking either biting the bullet and tinkering with some ARM SBCs, or reinstalling everything on a hypervisor (which ?) and booting up a PFSense VM.

Hi everyone.
I’m a junior devops with some professional experience as a sysadmin.
I love self-hosting and this weekend, I made some progress installing kubernetes on my server.
Got a bunch of things set up, argocd, cert-manager, etc.
And then I thought it’d be cool to have a couple more extra features, more on the networking side, like a VPN to access my home lab remotely, a DNS and DHCP server… nothing a reasonable firewall like PFSense cant do.
But that being said, I have been struggling to find a fitting solution.

My problem is, all the hardware I have at my disposal are my one bare-metal server, running ubuntu and kubernetes, as well as a couple raspberry PIs and an espressoBIN (an ARM-based router/firewall SBC).
The EspressoBin can, at best, run OpenWRT, or if I’m feeling very adventurous, I can try porting OPNSense to it (PFSense images for ARM are not available AFAIK). In both cases, there would be serious setting up and tinkering necessary, and it would be hardly reproductible and scalable, in stark contrast with how powerful and fluid my production flow with ArgoCD is.

The other option I came up with would be to install a hypervisor on my server and run kubernetes nodes on that. With that, I would keep my productiveness and scalability (and even improve the latter), and be able to boot up a PFSense (or OPNSense) VM. However, I then wonder about the hypervisor : which should I use ? I have some experience with Proxmox, and i’m not trying to do anything fancy so it should work just fine… but the Proxmox devs don’t seem willing to do any integration with Kubernetes on their platform. That might not end up being a problem, I just don’t know. That, and I’m not sure how I would get my storage working for optimal performance (I’ll get more into details about that if necessary). In my experience, there’s no way to keep a flowing experience for very long with Proxmox, but that might just be me.
I would rather always use FOSS, and another option i’ve come across is Citrix, with which I have no experience and have barely just heard of before. It apparently has decent Kubernetes integration, but I’m not sure to what extent that’s going to be relevant for me.

A third option would be to just put some money on the side, and get a proper router (either a small x86 SBC or an actual cheap router). Considering the primary use for this whole ordeal would be access for when i’m on the go, and how rarely I actually AM on the go, I’d be ok waiting to get a more productive solution.

I’d like to know if you have ideas for how I could set up a productive networking environment for my home cluster. If you know something I don’t, I’m all ears !

If there is some interest for how I did all this by the way, I might make a post about it : ).

2

A server is a function, not a hardware spec. Meaning that any machine that serves its data to the outside world is a server, even if said machine doesn’t resemble the traditional 19" rack-mounted case. So, get yourself an old(er) laptop from the usual suspects (Lenovo, Dell, HP, Sony, et all) and use it as a firewall/router/DHCP server. Added benefits: build-in UPS capabilities and a local screen+keyboard for troubleshooting :wink: Replace the old 2.5" HDD with a decent SSD for speed, a 256GB model is plenty for the OS and any VM’s you want to run on it. If you need more storage, the 1TB range appears to be the sweet-spot on price/capacity these days.

HTH!

2 Likes
3

Thanks for your reply, though this isn’t quite what I asked. I’m well aware of the ambiguity of the term “server”. That’s not where my issue is.
I cannot get new hardware for the time being. In fact, I wouldn’t be asking for help if I could.

I’m looking for a software solution to elegantly integrate networking into my kubernetes & argoCD flow.
This has more to do with, “what solutions are there that would help me be proficient”, rather than, “How do I install a firewall”.

4

Just spin up Proxmox with a few ubuntu/debian VM’s or do some alpine LXC with docker/kuberneties on them.

You dont need multiple hardware bits, you just need multiple VM/Containers.

5

IMHO kubernetes is not the best fit for very infra level things like DNS or DHCP at home.

What I’ve done at my home is docker with docker-compose, and this is because:

  1. Everyone has an example compose file online for some service they’re working on - good starting point when you’re running stuff
  2. I don’t have many hosts or distributed storage so any redundancy between the host that hosts my stuff and something else will need to happen with cooperation of the app. Specifically, I can’t just restart a DHCP server on another host pointing at NFS when NFS server is dead.

I have a directory (git repo) with configs, and I have a docker-compose.yml file at the root, and a few subdirectories with configs for each one of the services that needs it, like dnsmasq and nginx and caddy configs and so on.

The trickiest bit was putting a DHCP server into a container, it needs a raw socket on an interface (unless you have a switch to DHCP relay), which means I need to either have “host networking” ie. no network namespace for this container, or I need to create some bridge, or a macvlan interface and then deal with IP address numbering. So, I’ve done macvlanv for it, docker manages other stuff, it works great.

Do you really really need kubectl/argocd ?

6

As people said before K8s is not really a good candidate for infrastructure network stuff.

However if you want to manage everything through kubernetes manifests Rancher offers a controller that can deploy VMs through Kubernetes API. That way you can have those pesky 90s services through VMs while you run everything else on Kubernetes. Have a look it might be the solution for you:

1 Like
7

This sounds like the sort of tinkering I would rather avoid (the whole socket-in-a-container thing). I don’t just want something that works, I would like something that isnt a slow down relative to everything else I’m doing.
The reason I use K8 and ArgoCD is that I’m extremely familiar with these tools, they speed me up like crazy, especially compared to how slow setting up anything felt when I had everything based on Proxmox.
Anything else is a slow down, even docker-compose, though I agree generally, compose is a better fit.
This is certainly an interesting solution though, but I don’t think it’ll work for me.

1 Like
8

This is exactly the sort of solution I wanted to know about.

I’m starting to see that. Kubernetes isnt built for networking at all (though that might come I suppose, as it has started to for storage, to some degree).

This isn’t bad. I’m going to look into it. Thanks : )

9

Just install LXD on your already running Ubuntu install. Then you can have nested containers and create more K8s nodes (workers or masters) inside LXD containers. Do another LXD container with Pi-Hole in it and call it a day.

For the network side of things, you could install libvirt and run a single *BSD VM using Virt-Manager (either locally if you are running a desktop environment / window manager, or from another Linux box, you can install virt-manager on another linux PC or VM or server and connect from that to your Ubuntu server to manage VMs).

I’d say to stay away from Citrix. As for a router, you can either get something that can run dd-wrt or openwrt, or make your own with either a x86 box (Protectli) or ARM SBCs (NanoPi R4S) or RockPro64 with a PCI-E network card.

You have Raspberry Pis already. My main router is a Pi 3. I use the WiFi as WAN and the LAN side has VLANs set on it. I run iptables as the firewall. You don’t need to use the WiFi (or you can use it to give WiFi to other devices, i.e. an Access Point). Grab a copy of Alpine Linux and follow my Easy to Follow guide, maybe adapt it a bit for your own infrastructure.

This wiki will guide you through all the setup steps, from DHCP to firewall rules. I don’t have the VPN setup, but I have linked PLL’s Wireguard setup in there. Also, Wendell just released a wireguard guide, I can’t wait to watch it.

Then you can install something like Unbound on the Pi and run DNS too (or maybe Pi-Hole, but I don’t know its support for Alpine.

You can either buy a USB to Ethernet adapter and have 2 ethernet ports on your Pi, or setup VLANs on the single Pi port. But keep in mind your speed will be a bit limited by your port (100 Mbps for Pi 3 and older, 1 Gbps for Pi 4). On Alpine, you can run ip link add link eth0 name eth0.70 type vlan id 70 && ip link set dev eth0.70 up to create a VLAN with ID 70 (assuming your switch supports it obviously). I just put this command in my startup script that I described in the guide.