I’d like to build an infrastructure for a Kubernetes cluster. Obviously, this can be done using a service like AWS, but I’m interested in infrastructure - so I’d like to do it on my own.
I’m aiming to achieve a host infrastructure which allows a Kubernetes cluster to do what it needs to maintain it’s desired state. For example, provision resources from the host infrastructure, including provisioning persistent volumes, creating and scaling containers, etc.
I’d like to stick to Open Source software on this one and so far i’ve found OpenStack and oVirt. These two appear to allow you to build a datacenter, attach resources, and then have APIs that Kubernetes can use to provision the datacenter’s resources. In addition, both have a WebGUI to quickly get an overview of the cluster.
Are there any others platforms out there that I might be overlooking?
Integration environment was about 10 physical machines in a small openstack deployment I used for personal projects, but the work justification was testing upgrades and whatnot. Often times, changes to the underlying system would cause a shitfest of breakage if you don’t know how to do it properly, so everything was LVM snapshotted so I could roll back to before the updates and test again, and again, and again, for i in [1..50] do; echo 'and again, '; done
We had one other environment and it held all our prod and dev environments. AWS calls them accounts, openstack calls them projects or tenants, depending on version.
My currently favored path to self-hosted Kubernetes involves the following pieces:
RKE: The Kubernetes installer component of Rancher. Requirements are very simple, basically just a set of nodes with Docker running and an ssh-enabled user account with write access to the Docker socket. RKE will automate the installation process and produce a cluster with a containerized Kubelet, high availability control plane, API secured with certificates, RBAC, networking (including cluster DNS), etc. This cluster is also very easy to purge and rebuild, since the Kubernetes components are themselves containerized.
MetalLB: This is a component you install into your cluster which provides a LoadBalancer service. In L2 mode, this is as simple as giving MetalLB a range of IP addresses in the same subnet as your nodes, and these will be automatically assigned to make services reachable from outside the cluster when a manifest requires them. No special network infrastructure is required.
Ceph: A Ceph RBD pool is natively supported by Kubernetes as a StorageClass for automatic storage provisioning to pods. The preflight checklist for ceph-deploy is a bit more involved than for RKE, but essentially you just need to be able to dedicate a block device for each OSD, and at least as many nodes as you want replicas in your pool(s), preferably more.
The reasons I like this architecture are:
If you can provide LoadBalancer and StorageClass resources, then a lot of Helm charts that “just work” in a public cloud will also “just work” on your own hardware.
The same software stack is usable both in virtualized environments and on bare metal. I know you’re kind of specifically asking about virtualization platforms, but a solution which works on bare metal will also work on any virtualization solution you would be likely to self-host, and it will be neutral about your choice. You could change the platform without having to learn or build new tools to deploy things into it.
You have the option to run it in a converged pattern, with the Ceph and Kubernetes control planes on 3 nodes and Kubernetes workers and Ceph OSDs on the rest. Combined with MetalLB this keeps the whole thing very self-contained; you retain high availability without additional hardware or external dependencies beyond what you needed for Kubernetes anyway.
Problems this architecture does not solve:
Management of external DNS or NAT: No mechanism is provided here to automagically get your services on the Internet like there might be in a managed Kubernetes in the public cloud. In the likely scenario that your nodes are running in an RFC1918 address space, you’ll still need whatever tool you use to forward traffic to your LoadBalancer services much like you would for VMs. This can be particularly problematic for container images with a built in Let’s Encrypt integration.
Auto-scaling of the cluster (i.e., dynamically adding or removing nodes): I believe there are some projects which can do this on VMware and theoretically you should be able to do so on oVirt as well, but I’m not aware of any pre-existing software for this. I haven’t put any effort into solving this since it’s not very relevant for my use case. However, you can still set up auto-scaling services up to the limits of the resources available to a staticly provisioned cluster. This is a built-in capability of Kubernetes which does not require any integration with the infrastructure provider. You can also add or remove nodes without interrupting services using RKE, ceph-deploy, and your existing provisioning tools. It just won’t be automatic in response to demand.
You might also want to look at the whole Rancher platform in addition to RKE. This gets you your dashboard, support for managing multiple Kubernetes clusters, and integration with external identity providers such as LDAP. If it supported the oVirt API for provisioning it would pretty much complete the FOSS solution you asked for, but I’m not sure if that’s on the roadmap.
For automated provisioning of nodes, check out Razor. It is associated with Puppet, but can be used standalone. It will work with any kind of node which can be set to PXE boot (preferably by default), which as per the above makes it neutral to your choice of self-hosted infrastructure provider.
