Noob Question: Why Run VMs?

Software & Operating Systems
, ,
1

I see lots of home lab YTers running proxmox, vmware, and other hypervisors to run VMs. But being a software developer, I’ve seen the switch from VMs to containers about 8 or so years ago (give or take), because VMs require a lot more resources and overhead (like os scheduling, anti virus software, other background services, etc) just to run a single program per container. Am I missing some kind of typical system requirement that one would want to run VMs?

Is it just that I don’t have these problems? Most of the time I run into problems that could be solved with a bunch of hosts and kubernetes + a nas.

4 Likes
2

Two reasons for me:

Security boundaries - I believe it to be harder to break out of a Xen domU than it is to break out of your average docker container arrangement

Separate kernels - it’s just more flexible, be it because you want to use a distro supplied kernel in a guest or BSD/Windows etc.

It’s interesting to look at what the large cloud providers do, eg:

  • AWS went with microvms for AWS Lambda & Fargate:
    https://firecracker-microvm.github.io/

  • Google apparently went with https://gvisor.dev/ for some offerings, which is a more security centric container approach where containers can’t directly make syscalls to the host kernel. You can setup docker to use gvisor, though few do

Security boundaries are, obviously, a major driving factor in public cloud technology decisions.

6 Likes
3

Addendum:

Did you see a switch from containers to VMs? Or did you see a move to running containers within VMs?
Eg: k8s w/ containers on EC2^

Even at the individual level of a cliche SV macbook carrying employee, they’re running docker within a VM.

^Baremetal EC2s aside, I believe them to not be in common usage

6 Likes
4

The bloat of added things VM’s need, are also tunabkes that they can have as added functionality.

More control over a whole OS, gives more modularity to what parts to run, and more ability to run them.

Containers are also used, but are much more limited.

Also, homelabbers often have smaller fleets, so more a case of a few individually customised VM’s, instead of many automated, scripted containers/services.

Some people bridge the gaps, like Jeff geerling, who goes from dedicated hardware (pi’s) even less capable than some VM’s, to servers running VM’s to orchestrated fleets of containers running via ansible (he even wrote the book on it!) well, at least one book. Also, he might use ansibe to script VM’s too/instead, so I may be over-hyping… But I think it was kubernetes clusters

2 Likes
5

Yeah, with leftover enterprise gear as a staple in the homelab, why bother with containers when a VM can have any customized OS do 1 or many things. I actually do not have any containers running anymore. Even though some of my shrunken VMs, like homeassistant, could be on one.

1 Like
6

Less interdependency. I can run a debian and a centos and a windows and a x y z without conflicts.

2 Likes
7

I saw it as migration from deploying software directly on ec2 instances VMs to running docker images inside VMs. Unfortunately, I don’t think the choice between running containers inside VMs or on a base os was an explicit choice by the developers (nor do developers care that much), sans the security aspect.

Definitely, ec2 uses VMs as the physical hardware probably can run many VMs in order to support different ec2 node types. But if my only need is to run containers in a way that is HA, scales, and is easy to use, then as a developer, I don’t necessarily care if the container is running inside a VM or on a host OS directly. So something like netboot to boot strap an os with the ability to run kubernetes or something, could also work. Even though I haven’t looked too deeply into it, managed kubernetes products could work like this (with a max on memory, ram and disk resources). Though this doesn’t necessarily take into account security concerns.

1 Like
8

I guess my counter argument here is why couple services? Let’s say you want to tear down one service but leave the rest, then I’d have to manually go and login to the VM tear it down and clean out the files that service was using. If you ever need to split services from an existing VM into multiple VMs across nodes due to resource constraints, I assume that would be pretty painful (maybe I’m wrong here). I think this also holds for merging multiple services from different VMs into a single VM. The final scenario, moving an existing VM to another host would would be easy.

The thing is that all of these are pretty easily handled when using containers, as they are already modularized to begin with, and moving them around is handled through something like kubernetes because of having to handle node failures (or maintenance) means, you can handle tearing it down and re-hosting it somewhere else. Also, when you couple services it affects scaling. For example, if I want to scale service A, but I have both A & B in the same VM, making copies scales both A & B when I really only want to scale service B.

1 Like
9

The kernel part here is interesting. I want to say that when docker uses ubuntu as a base image, the library calls to the os wind up getting translated to host os calls through the daemon, but I’m not 100% sure how that works.

I’ve heard that jails in BSD are really nice, but are a somewhat similar concept to docker.

1 Like
10

I definitely agree there is added functionality of running windows vs linux especially if you are running C# stack and need to make windows specific library calls without emulating them. This really rears its head with gaming and DX12.

More broadly speaking how does it give you more modularity? Are we talking being able to run multiple services side by side? Are we talking which processes are constrained to which cores?

Ansible is amazing. I eventially got sick and tired of manually setting up my new mac every single time I switched companies or bought a new one, so I setup some ansible scripts to set it up for me, provided I setup python & ssh first. Definitely something I wish I had done sooner.

11

i do ‘things’ not ‘apps’ with VM’s.

my ‘WEB Services’ VM has the most stuff on it. it is a Debian VM that has 4 websites hosted, Emby, and Nextcloud. but the storage size of the VM itself is only 120GB for the boot OS and all the logs and the Databases for all that stuff. This VM was 8 years old, still based on BIOS in the VM but was running current Debian 11 via fairly rugular Apt upgrades.

i wanted to add a T1000 GPU i had laying around to Emby so that it could hardware transcode instead of using CPU resources.

I made another VM with Debian that booted via EFI, passed through the GPU, loaded the drivers, bare MariaDB, Emby, and Apache. Rsynced the sites and Emby cache, added the storage from the NAS in FSTAB, installed and reconnected Nextcloud to its pool and my AD server. done.

so lets break this down. could apache (or even each site) be a container? absolutely.

Could Emby, my Database host, and nextcloud, all be on containers? 100%

but, um, what would that gain me? it would add another layer of management for stuff that rarely needs any attention other than updates. i would need to update potentially multiple containers, or some container template, and deployment.

but what if an update to Apache breaks 1 website or does something unexpected? i just grab the last working VM from backup and load it, takes like 5 minutes and EVERYTHING works fine again. and this stuff does not change independently enough for a days lost consumption to matter.

And Apache virtual-hosts does any seperation for the sites that a container could do. plus i do not need to utilize any of those weird reverse proxy systems or dynamic DNS forwarders or load ballancers or blaahh blah blahs. i have proxmos connected to the network VIA a trunk port with VLAN support, my AD does permissions to the NAS, Users access the network with their credentials, and even if an attacker breaches this Web Services VM he has Read Access to all of the media that i allow access to for this server to begin with.

not to mention Crowdsec and other things on prem.

containers would be 1 more thing to managae, and gain me nothing that i have found useful.

this is just one instance of containers not creating a savings. my AD server and home assistant are single TINY VMs that i RARELY touch, sometimes the kernel on the AD server is down right old before i do anything about it. but this VM has passed through hardware and very specific network and software config that would be a pain to replicate via a container.

3 Likes
12

Huh, Literally didn’t even think of Windows in a VM.
But I know people run Windows Server, in VM’s.
Then they can move the server to a different host/hypervisor if and when needed, easier than bare metal.

The modularity, I was thinking more linux/BSD, with stripped down drivers, no GUI, and switching whole networking stacks, if the apps running work better with NetworkManager or SystemdNetworkd or whatever.

Obviously Windows could have a load of un-needed software stripped out too, to reduce the bloat?

13

Personally I chose containers over VMs any time unless I need something that only VM offers like a different OS. All server software I use at work and home runs on Linux.

I have a MacOS vm on my macOS laptop to isolate the trash that is Citrix workspace from my untainted OS. The infrastructure I connect only allows windows and macOS due to some bullshit with app protection. Seriously hope that software burns to death in some fund management dungeon.

I also believe the threat model of most people is not high enough for the separate kernel justification. Most recent security blunders are hardware stuff and vms won’t save you from that.

1 Like
14

[edit, covered by Zeddicus]
It can also be easier to pass hardware though to a VM, than a container

15

There’s a large rogue’s gallery of projects that aim to “debloat” windows in various ways, ranging from post install cleanup scripts or generation of minimal ISOs to “hey dumbass install this custom ISO it’s totally safe and gaming ;DDD”. The problem with many of these is they tend to carpet bomb the security features.

However, if you are still on Windows 3.11, here’s a useful guide that can make it fit on a floppy disk: http://www.geocities.ws/politalk/win31/slimwin.htm

1 Like
16

I don’t think that’s always correct. Rather it depends.

In that example with AMD cards you just need to map /dev/dri/* to the container instead of doing all that shingamabob with uefi and mapping PCI IDs. I run most ROCM stuff from a distrobox rootless container.

I suspect nvidia is just as easy since they do a big deal about their integration with containers. AFAIK its just about having the driver and passing the gpu capability to the container.

1 Like
17

was thinking of networking gear and storage controllers and such as well.

Buuut, I could be wrong.

Might be real simple to pass hardware to containers too

1 Like
18

Thats true. I don’t know how to pass direct access to a network card without using bridges or macvlan. There is host network but this is another thing. GPUs are easier though.

I imagine this would be to run a virtualized router firewall?

2 Likes
19

so what i came up with was the host should support virtualized switching (open vSwitch) for example. and then trunk the vSwitch to the router (virtual or real) then connect any container to the vswitch. (could use a virtual bridge to create VLAN ports, connected to ‘ports’ as needed)

again, containers just end up being more work for my home lab.

20

This kind of passthrough has the disadvantage of being locked to host’s drivers, which may become incompatible with container runtime - better to keep them in sync. Also if you crash driver you may need to reboot the full host instead of just the VM.

PCI passthrough should be really easy though. As long as you’re not doing hotplug a simple <hostdev mode='subsystem' type='pci' managed='yes'><source><address {src_xml_pci}/></source></hostdev> should be enough.

You can do really slimmed-down VMs using small distros such as Alpine.

I, personally, am not a fan of virtualizing everything, e.g. I much prefer WSL1 (native syscall translation) over WSL2 (virtualized kernel) even though the latter supports more features. But I can see benefits (of VMs over containers). E.g. with containers you have much lower control over the filesystems - I don’t think you can even do bind mounts without adding extra permissions. Meanwhile with virtualization it’s easier to pass full block devices or even full hardware (e.g. SCSI controller) and have full control.