So my current setup is running docker Natively on Proxmox and its ZFS file system setup.
I did try a while ago trying to get GPU passthrough to Docker on an LXC container, and could always get the LXC passthrough working, but never to Docker. I finally had a break and found the missing link (edit the toml file to set no-cgroups = true), and now docker can happy use the GPU.
I have however seen some “concern” about docker in an LXC and was wondering how “real” that is, or just people not liking virtualization in virtualization.
Reason being is I am thinking of moving all my current docker containers into multiple LXC containers to keep them more separated. Also, to stop an annoying issue that stops me upgrading some containers because ZFS locks it sometimes until I reboot the entire machine. Happens more often than I would like tbh.
Almost all docker container images are oci(open container image) compatible and lxc containers are also oci compatible(most lxc images are oci compatible also). So running docker in lxc or lxc in docker is completely unnecessary, there is almost zero reason to do this, just run the image in lxc if your having issues with docker. There is a few fringe cases of lxc or docker handling something better than the other, but in those cases your probably not looking to run one in the other. lxc in general are heavier containers than docker containers.
Just like you can run a vm, in a vm, in a vm. You can also do the same with containers, but there is rarely a “good” reason to do so outside of “because i can”.
Also, if your going to put containers in containers, since you can run docker containers in other docker containers just like you can put docker containers in lxc containers, why not just put the docker containers in other docker containers, and stick with one container platform.
Lastly is the point that since proxmox is built around lxc why not just move all your docker images into lxc containers, and be done with docker.
P.S. LXD is not LXC they are different container technologies that work differently, and LXD containers specifically are not oci compatible.
Funny I just tried this yesterday and it’s only partially true / a bit misleading.
Yes you can reuse the rootfs of a docker container as lxc rootfs, but that does not make the lxc containers function like (most) docker containers.
We are missing all environment variables, excepted mount points and you will have to set cmd/entrypoint to run at startup (depending on the setup this might need to run this as pid 1).
I also had problems in alpine containers. The eth0 was down and needed to be enabled using ifup eth0.
So tl;dr, this might not be that easy.
If I understand your intention right you want more security, if that’s so, I‘d rather look how to run this app directly in a unprivileged lxc container.
Firstly reasons for running in Docker and not direct LXC (long time ago I used to run services like this).
Updatability: With LXC (in the past), each one was manually made and had to update manually when the application/s changed. ie. the “arr” software and Emby/Jellyfin etc. With docker, running Watchtower, set it and forget it.
Reuse: I have everything already running in docker natively on the proxmox box, which I know is not a “good” practice. It also does cause some issues with updating sometimes and the ZFS dataset sometimes becomes locked and the only way of “unlocking” it that works so far is complete reboot of the entire box. By having those containers stored within the LXC Container itself, it wouldn’t be using docker with the ZFS driver.
Ease of Use: All my folders etc is already stored on the zfs file system, so would just need to mount them in the various LXC’s and no real other configuration required. A quick edit of the compose files to point to those shares mounted in the LXC and done.
Portability: I am also running an OrangePi5 and another miniPC that are on debian just running docker. Usually the lighter weight stuff is on there and stuff that doesn’t need access to the mass storage on the proxmox box. Sometimes I do move docks(?) between the systems.
So yes, while I know I could run all these apps directly on the LXC’s themselves, it just feels like adding a whole bunch of extra unneeded work. This is a homeserver with NginxProxyManager on the front, so no ports are open besides 443 and 80.
As for running a docker image as an LCX, never heard of that, but then wouldn’t I still have the issue of making it difficult to update? But tbh, this is why I am asking in this forum to see if there are better ways to do what I am thinking of doing.
I did not know about this… I Will have to look into it and see how it can fit in with my infrastructure.
we are talking about LXC not Virtual Machines. A virtual machine would consume the entire GPU through passthrough. There is no sharing on it, so everything would need to be done in that single VM. A LXC can share the GPU amongst other LXC’s so there is no exclusive locking of it.
Yes you might have to be more selective about where your containers are running instead of just having one big pool of everything. I don’t really see the difference in a big pool on the host to a big pool in a VM with hardware passed through to it except from maybe the networking side of things, but that’s just me
lol edited my last post my dyslexic brain kept typing lcx instead of lxc and i did not catch it. Sorry for the confusion
Lxc does support env variables just pass them during creation tiime using — env var=value. this would add the env var to your container config for you or you can manually add the to the container config later.
Ok so lxc containers sit somewhere between vms and docker containers as a virtualization platform, why they are heavier. As such they require more configuration since they can be more powerfull. The docker dameon does a lot of heavy lifting on the networking side of a container that lxc does not do by default nor does proxmox. You have to setup and run the a bridge network for lxc containers, and configure your container by editing its config file to use that bridge network. also for some reason the lxc.init.cmd is not always set when importing an docker oci image, its usualy /init so you just add lxc.init.cmd = /init to your lxc container config file usualy found at /var/lib/lxc/<containername>/config. the last piece of the puzzel is becuse docker usually handles networking for the container usually i have to edit the /init file in the container and add udhcpc -i eth0 to the in afther the addpath stuff and export but before the actual exec to have the container request a ipaddr from the bridge interface to get an ip.
So for instance to start an Homeassistant container assuming you have your lxc bridge network adapter setup
eidt /var/lib/lxc/<containername>/config to make sure the container is using your bridge network addapter and that the lxc.init.cmd is set correctly to what docker calls to start the container, for homeassistant its /init
start the container and attach it so i can edit the /init and add udhcpc -i eth0 so the container gets an ip from the bridge
restart the container now homeassitant should have lauched correctly on your network with the ip aquired via dhcp
this is a simplified example, in reality you would probaly have the container get the same ip every time not just take a whatever the dhcp server of the bridge handed out. But it totaly works and takes less than 5 minutes to setup a contiainer.
After doing this is when you realize how much work the docker dameon is doing for you and the power of docker compose.
However, it does seem like a lot of extra work, whereas I can fire up an LXC, ssh into it. Fire off a few composes and be done.
The method you are describing sounds like every docker container would have to be a separate lxc. This would be especially painful for a compose of immich which requires 4 different containers.
In the end, I really want each LXC to run a number of docker containers grouped by their function. ie. all media players in one container, downloaders in another container that has a VPN set for communication.
But again, thanks for the information I am learning a lot of interesting things that I was unaware of.
I got side tracked from your original issue id actually like to help you with your original issue. My opinion on virtualization in virtualization aside, there is nothing strictly speaking wrong with what your doing, wanting to put groups of docker containers in lxc containers.
Id love to here more on this, see if i have any thoughts. Guess im not seeing how lxc is going to solve this issue. Is your docker install on proxmox using the zfs storage driver?
Its not often, but if something receives multiple updates in a short timespan. ie. I could be tweaking a container etc. the dataset can sometimes get in a locked state. Tried many solutions online searching for what is locking it… and no joy. Rebooting entire system is only option.
If I used in an LXC, then it should default to overlay because inside the LXC would not be zfs (if i am correct from what I have been seeing). Granted I have heard of LXC’s having the issue sometimes, but hoping not to the extent that the docker zfs driver has them.
EDIT: All Actual data would be on mounts to the LXC from the zfs system. so nothing of any value would be stored in the LXC itself.
well its more how zfs is used the docker zfs storage driver is using a datasets and snapshots to basicly do its layering to save space and not rewrite data. when runing docker in the lxc im gussing docker there is not using the zfs storage driver if you check the json.
If so, which then brings the question have you changed back to the regular non zfs storage driver on your host to see if the zfs problems go away(the images and docker can still be on zfs dataset that you manually snapshot for backup replication purposes just dont give docker control). for the test you would need to use a new dataset as the root dir for docker and its images/data
Nope, I followed the “best practices” on the docker site in setting it up on ZFS storage. I haven’t checked, but I do recall seeing some warnings against using the overlay storage on zfs. At the time I had only really run docker in a Jail on TrueNAS scale (they were hyper anti-docker at the time trying to block every avenue of use) and that was set to use overlay2 system. (docker not jail. Jail obviously ran on the ZFS). Then often had issues on TrueNAS trying to run VM’s … some patches would work, then they would release another patch that broke the VMs.
TrueNAS is for storage, trying to use their other features was a nightmare imho.
So with that pain, I decided to go back ro proxmox for a solid setup and for when I needed the occasionaly VM to test stuff.
Sorry went off ona tangent there. LOL
EDIT: But lets put it this way… move to LXC is not really about the ZFS thing… that is more of an irritant that would hopefully be solved.
if you are using Docker I think there is no reason to use LXC
in real life Docker almost always runs bare metal, and doing so makes the GPU enablement (because you dont “passthrough” a GPU into Docker the way you do in a VM) super easy
i think your missing what im saying, docker regardless of what storage driver your using all do an “overlay system”.
With the zfs storage driver there able to do this via datasets and snap shots. You would not to my understanding be able to do this with the zfs storage driver in a container. So my question is whats the backing store of the docker daemon in the lxc containers and what storage driver are the lxc docker daemon’s using because i doubt its the zfs storage driver. The best option if the lxc is on a zfs dataset is the vfs storage driver.
There are multiple storage drivers the easiest AB test is probably the vfs storage driver. If your having issues in one place and not the other, its probably coming from whatever the difference is.
I have just set the storage on the current native docker on proxmox to overlay. Had to do create a new set and redownload all the images etc. but it should stop using ZFS datasets to store the images and just rely on the main dataset they are all stored in.
Hopefully this will eliminate the locking issue. Worst case is that I will go back to what it was.
I am still trying to decide the benefit/cost of moving them all to LXC containers.
