Greetings. I’ve enjoyed the Level1News for a while, and this is my first foray into the forums. I’ve used virtualization in one form or another for a number of years (decades in fact), from VMware 1.x server and VirtualPC back in the day, to Hyper-V on Server 2008 and VirtualBox on both Linux and Windows 7/10 hosts, to my current setup of ProxMox on repurposed server hardware in my home lab.
The time has come to upgrade my current home lab (dual Xeon L5630/Supermicro mobo/48GB DDR3/5x2TB WD Red HDDs) to something more modern. The question comes in what virtualization system I want to replace ProxMox with, or if I want to keep that. Some of the VMs I use, seem like they would be good candidates for containers as opposed to a full VM - running BIND, LAMP stacks including Nextcloud, and a controller for my Ubiquiti WAP. Others would likely still require a traditional VM, in that I use VNC to connect to Ubuntu-based desktops for different tasks.
The catch is that the idea of containers is still relatively new to me, and the resources I’m finding on Google don’t seem to quite answer the questions I have. So, all of that said, does anybody have any input or suggestions on containers vs VMs vs hybrid hosts (if that’s even a thing?), or what virtualization/container host to use? I’d prefer Linux-based options as I have taken courses on and have a basic understanding of RedHad/CentOS (but I’m not crazy about the changes coming there), and have a more hands-on practical knowledge and understanding of Ubuntu/Debian.
My thought is to build two relatively identical hosts that interface with the rest of the network over the gigabit infrastructure I have in place, and a dedicated 10-gigabit backbone between them to facilitate data sharing over (most likely) NFS. Each machine would have its own load as well as serve for data backup from the other.
Any insight or feedback would be appreciated. And if this topic needs to be moved to a different thread, please advise.
The method I currently use is to run VSphere on my R720s and use the “Expose HW Assisted Virtualization” option under the CPU settings for a VM to setup a Docker host VM. That has worked for a few projects of mine like Nextcloud and a private wiki, but I haven’t really tested the performance of them.
My understanding is that there isn’t a ton of performance loss anymore to having your Docker containers nested in VMs, perhaps I’m wrong on that. I have considered it just being a stepping stone to having a dedicated Docker host, or using a hypervisor with VM and Docker capabilities. But until I run into an issue with nesting I’m leaning towards that. Easier for me to manage for now.
Studying for the LFCS exam, the introduced me to LXC/LXD for hybridized containers and I quite like that more than Docker, Snaps, and full blown VMs for stack development and distribution.
Basically an LXC is a light weight VM that uses existing system resources and bindings so that it does not have to run it’s own. In that way, it is more like a container but has the added bonus of having it’s own full blown OS. So you can literally ship a system that can be updated, which is great if you do not trust your customer or developers to be able to keep their actual environment updated. Caveat is that this only works with GNU/Linux systems and some *unix like systems that have a GNU compatible userland.
I am used to using Vware, KVM, and VirtualBox (VMs) and "Docker"and SNAPS (containers) in enterprise environments but I have never run into LXC or JAILS. JAILS is a BSD technology that also looks pretty promising but is more VM oriented for non-native system.
I can also only recommend LXC.
I’m currently running a setup of LXC and libvirtd on Debian 10.
I’ve gotten into the habit of just writing bash scripts for everything, which has resulted in some nice scripts which I have published here.
There is also a script for easy testing of such a system using libvirt.
I can setup a new host system for both containers and regular libvirt-based VMs in less than 2min:
(Video also demonstrates the apt-cacher-ng server that is automatically setup, as well as log forwarding. My terminal emulator weirded out out at the end there.-.)
Most of it is commented.
I’ve previously also used LXD on Ubuntu, but because I prefer The Debian way™, (which includes not installing snap packages because they are annoying) this setup uses LXC on Debian 10.
I can make a short post about all the stuff these scripts do if there is interest…
I love LXC but it just does not have the same tools as docker/containerd. If all you want to run is a single host with a few services on it, then its fine to great. But if the goal is to build a multi service cluster with HA failover and live migration you just cant do that with LXC.
It seems like the industry has move forward and developed around the docker concept and LXC will live on but as a second class container option when it comes to support and tools.
So im not saying dont use it, just make sure that your needs match the container system your choosing.
First, for a home lab high availability is just not a concern.
Also there is more than one way to archive high availability, most of the setups that I’ve seen just “manually” setup more than one machine the same way, and use clever networking(e.g. DNS, multicast). No need for a hypervisor to get involved.
Dont assume anyone’s needs or goals HA may not be a requirement for most homelabbers but plenty of people want to play with it or learn from it which means it is a concern. Also someone wanting to do development work may want to test their code under a production like environment.
I personally want to build a full HA home cloud, so for me distributed containers with orchestration and distributed block storage is essential.
After seeing the comments here I am definitely planning to look into LXC to see if it offers a better setup for myself. I do put a pretty high priority on high-availability for my home lab since I run some critical personal services through it for myself and some of my family. Specifically my password manager is desperately important to have access to remotely anytime I need it, and a home lab setup for that means I don’t have to pay a monthly fee and feel more in control of it.
Adoption of these tools was slow for me because I was plagued with HW/SW issues causing the services to go down until I had time to tinker with it on the weekend. Once I had a decent set of tools I was happy with I’ve spent more of my time working on automatic backup, quick redeployment, and improved access to them.
Containers sit in between chroots and VMs. They’re kind of like FreeBSD jails or Solaris zones. But they’re a lot more popular and there’s standard packaging and redistribution mechanisms for images - Docker. They also organically separate software/code from data. You don’t keep configs/data in containers you keep it on the host or in some distributed storage and you spin up containers pointing at the config/data.
Run in containers instead of VMs whatever you can - it’s much lighter in terms of resources and easier to use Docker to build and package stuff !standard format).
When it comes to running them, if it’s one or two hosts only or development on a laptop or similar, use Docker or Podman. Two to 50 hosts, use one of the light weight Kubernetes-like offerings like Rancher. More than 50 hosts in your lab ($1M+ hardware) use Kubernetes proper.
Some folks who want to learn things here, take 1-2 hosts, split them into 5-10VMs and start setting up Kubernetes from there. Things that differ with each step of scaling up are
How you manage incoming network traffic
How you manage network traffic between containers
How you manage persistent storage
On a few hosts at home, you just map some directories into container paths, and have bridges and VLANs.
On the far end you have Rook that helps you run Ceph and that lets you provision file/volume storage, or you get to use it directly as an object store. And you have Istio and Calico on the traffic part of it, and you can grow this setup. Once you’re at $10M/year hardware spend, you should start negotiating with hyperscalers for public cloud stuff. It’s a thick margin industry, discounts for that level of spend can be substantial. I don’t think this kind of lab fits your home lab budget - unless your last name is Disney or Walmart or Amazon or Microsoft or Google where folks do spin up clusters like this for project work and testing stuff out.
Jeff Geerling, and Technotim make easily digestible video content on containers and homelabbing ; give it a glance.
Word of warning would be to take that stuff out of the lab and put it into a production environment so that you do not clobber anything that you rely on.
The lab should be used for testing and development. Best practice is that none of your productions stuff should ever live permanently in a lab environment.
HA may not be a requirement for most homelabbers but plenty of people want to play with it or learn from it which means it is a concern. Also someone wanting to do development work may want to test their code under a production like environment.