I have a proxmox cluster that I mostly use to stand up home lab servers. I’m a software engineer… Wondering whats the best way to roll kubernetes or docker on a proxmox cluster?
I have dual E5-2690v2; so I think I have 2 numa nodes per cpu… The largest my vm could be is 10 cores, before it’s larger than the numa node. Anyone speak to strategy here?
I probably want at least 4 workers, for testing and failover. (I have 3 nodes)
if docker containers are the only thing you plan to run on the cluster, why use proxmox?
just wondering.
I’m guessing it’s because
So, OP, your best bet on Proxmox is probably to run however many Ubuntu VMs you need and install microk8s and portainer (it’s the easiest to set up). If you want more efficiency, you can use LXC containers, however, the process is a little more involved and you can’t use the default Ubuntu installation method for microk8s, because snaps need systemd and other privilege options and the containers don’t have that (by their default secure settings).
However, if you want LXC, it’s probably easier to spin up CentOS containers and installing k3s there. It’s still a little involved due to some container inception (OCI containers in LXC containers), so you have to set up some stuff. Again, simplest, but not the most efficient is Ubuntu VMs with microk8s and portainer microk8s addon.
If you go with VMs, I wouldn’t allocate all 10 cores to 1 VM, but would make multiples of 4 (2 cores with 4 threads) per VM. So you’ll have more VMs, which means more nodes (3-5 master nodes, rest worker nodes). And if you can, try pinning the CPU cores.
I’d like to suggest sth as well. K0s recently got really really good and with k0sctl handling cluster lifecycles has never been easier! So I’d suggest creating 3 LXC 20.04.2 Master nodes with nesting and keyctl enabled and however many worker nodes you want that way you’re lean on resources and will have the best experience since the worrker nodes won’t be held back by lxc. Due to the control plane isolation set by default you won’t be able to schedule anything on the master’s (for better or worse) as they won’t even by shown in a kubectl get nodes. You can of course change this behaviour if you want, but I think it’s a pretty neat feature.
Also if you don’t necessarily need to run Proxmox, KubeVirt might also work pretty good.
Regarding storage, that’s the tricky part… If you have a ZFS storage (aka TrueNas, Ubuntu) you can try the democratic CSI. Else NFS provisioner is fine, but not great. If you don’t want that Longhorn is also a great option!
To auto create rancher managed kubernetes clusters on Harvester:
https://docs.harvesterhci.io/v0.2/rancher/node-driver/
I actually use Talos on Proxmox and it’s great! 
No, it’s really straight forward if you don’t have specific requirements for the host OS.
The only thing I didn’t do is deploy k8s with a highly available Control plane. This is totally fine normally but with Talos it’s pretty much a requirement. Without it you can’t upgrade your cluster to a new version since it completely wipes the node including the etcd cluster and then rejoins the cluster with the new version.
At least it won’t try to do that if you don’t have enough master’s XD
I might make a guide if anyone’s interested.