Suggestions for a Proxmox Cluster used for Daily Driver OS, Gaming OS, and k8s Nodes?

I’m wondering if anyone here has experience using Proxmox to run their daily driver OS, or if they have any advice, suggestions, or gotchas they’re willing to share. I’m aware of the virtualization tax, and I’m willing to pay it for the functionality I have in mind.

For context:

My plan is to have all the PCs I own using Debian base + Proxmox, cluster them together on a internal network. One hypervisor will be designated as a NAS using ZFS (SMB shares, VM/snapshot storage). Some of the guests will be exposed to the internet for web hosting/development. So, the plan is to have two networks, a “risky” network that has guest VMs exposed to the internet, and a “home” network that’s able to access the “risky” network, and the NAS shares.

I develop/use containers quite a bit, so I’m planning on deploying k8s/k3s guest node VMs to get a self-hosted/automation/IoT-gateway thing going (ideally replacing Github/Travis with Gitea/Drone or Jenkins or something). I’d go k8s bare metal, but most/all of the PCs are going to be running VMs with GUIs that need to be accessible on boot (there’s more than just me using these computers for stuff).

I currently switch between Ubuntu for work/software development/some gaming, and Windows for gaming (both on bare metal). Then, if/when the PC is going to be used for single-user PC things (using an OS with a GUI, specifically), scale down the resource of the other VMs… Or just provision what’s needed from the get-go.

Of interest, I’m using an Raspberry Pi 3 B+ as a router/gateway/firewall with a cheap unmanaged switch and USB NICs, but that shouldn’t matter as all it’s doing is DHCP provisioning and segmenting the networks (the internal network and the “home” network) as I’m planning on doing DNS later . My uplink is nowhere near saturating the 300Mbps speed limitation of the USB->NIC link of RPi).

Again, any thoughts or opinions are appreciated!

What’s the point of the cluster in this? Are you aiming for HA in proxmox or is there another reason?

I know people do something similar with unraid so it doesn’t seem too crazy to do with proxmox.

You can do this with pfSense and VLANs.

Just use gitlab with gitlab-runners. They’re available from the official repositories. Can configure them to use how you wish.

A little bit of HA, but that’ll probably be more in k8s scope (sorry for being vague, still trying to figure some things out). Ideally, the plan is to migrate or clone workloads/VMs between different HVs to make redeploying stuff easier and creating a more ‘virtual’ desktop across the house. Or something like that.

Yes, that’s true. VLANs are easier to implement too! :wink: But, I’m trying to get some experience working with the Pi in a non-standard use case (building out a Docker-based router that doesn’t mess with the “host” OS IP tables, specifically), and I’m also trying to make sure the gateway is locked down/as secure as possible (avoiding VLAN hopping, using separate physical NICs for each network segment to enforce separation and other reasons). Last time I looked, pfSense wasn’t battle hardened with ARM architecture, so I haven’t looked into it further. If/when the router is virtualized, pfSense seems to be the best bet.

I’ll look into this. Gitlab does seem nice, but wasn’t sure if the amount of stuff included would increase overhead… But it is an all-in-one… :thinking:

I’m not questioning your methods, I dont actually know that much about it myself. Sounds neat.