I run a small server with Proxmox, and I’m wondering what are your opinions on running Docker in separate LXC containers vs. running a specific VM for all Docker containers?
I started with LXC containers because I was more familiar with installing services the classic Linux way. I later added a VM specifically for running Docker containers. I’m thinking if I should continue this strategy and just add some more resources to the docker VM.
On one hand, backups seem to be easier with individual LXCs (I’ve had situations where I tried to update a Docker container but the new container broke the existing configuration and found it easiest just to restore the entire VM from backup). On the otherhand, it seems like more overhead to install Docker in each individual LXC.
Regardless of VM or LXC, I would only install docker once. There’s generally no need to create multiple docker VMs/LXCs on the same host. Unless you have a specific reason; like isolating outside traffic by creating a docker setup for only public services.
Backups are the same with VM or LXC on Proxmox.
The main advantages of LXC that I can think of:
- Slightly less resource overhead, but not much (debian minimal or alpine VM is pretty lightweight already).
- Ability to pass-through directories from the host.
- Ability to pass-through hardware acceleration from a GPU, without passing through the entire GPU.
- Ability to change CPU cores or RAM while it’s running.
I use individual lxc for each docker compose so I don’t have to revert 8 services at once if I need to restore.
I would also argue that an alpine lxc runs in 22mb ram by itself … Significantly smaller footprint on disk and in memory. But most importantly, lxc can actually share memory space effectively, one doesn’t need to reserve blocks of ram.
You don’t have to revert 8 services, you can stop/start them independently: docker compose stop <service name>.
This is actually how I update my services, I just stop the ones I want to update, pull, and restart them. I do them one or two at a time, mostly to mitigate issues. The same is true for pulling down new versions, my process is:
- edit the docker-compose file to update the image version(s) (e.g. from 1.0 -> 1.1, or 1.1 -> 2.0); I check changelog/release notes to check for any manual upgrade notices
- pull new images (doesn’t impact running services)
docker compose up -dbrings up any stopped services using new image(s)- test
- go back to 1 until all services are done
I do this whenever I remember, and it works pretty well.
How do you handle backups? Install restic or whatever in every container and set it up? What about updates for the OS and docker images, watchtower on them I imagine?
It sounds like a ton of admin overhead for no real benefit to me.
I just snapshot the parent lxc. The data itself isn’t part of the container at any level, so if I bung up compose yml or env, I can just flip it back. The only real benefit is that all my backups are in the same place in the same format.
Like I’m not actually opposed to managing docker in one unit, I just haven’t got there yet and this has worked so far.
If I were to move to a single platform for several docker, what would you suggest? For admin and backups?
What’s the purpose of running container in a container? Why not install docker on your host machine?
If you do that, Docker is stuck on that host. If it’s in an LXC it can move to another host. Plus, backing up and snapshotting are easier IMO.
Snapshotting in docker is as easy as docker commit. After that you can back it up with docker save. Then move to another host, but not without downtime.
However normally you need to backup/move only volumes attached to containers. If that’s not the way how you like to organize your services, you likely don’t need docker.
Dockers ‘take-over-system’ style of network management will interfere with proxmox networking.
I don’t use proxmox, but it works absolutely fine for me on my regular Linux system, which has a firewall, some background services, etc. Could you be more specific on the issues you’re running into?
Also, I only really expose two services on my host:
- Caddy - handles all TLS and proxies to all other services in the internal docker network
- Jellyfin - my crappy smart TV doesn’t seem to be able to handle Jellyfin + TLS for some reason, it causes the app to lock up
Everything else just connects through an internal-only docker network.
If you’re getting conflicts, I’m guessing you’ve configured things oddly, because by default, docker creates its own virtual interface to explicitly not interfere with anything else on the host.
A couple posts down explains it, docker completely steamrolls networking when you install it. https://forum.proxmox.com/threads/running-docker-on-the-proxmox-host-not-in-vm-ct.147580/
The other reason is if it’s on the host you can’t back it up using proxmox backup server with the rest of the VMs/CTs
Honestly you can do either.
LXC
-
shares host kernel (theoretically lighter weight)
-
less isolation from host (less secure)
-
devices are passed via device files
-
less flexible due to dependence on host
-
no live transfers
-
filesystem shared with host
virtualization
-
has own kernel and filesystem
-
supports live transfers
-
hardware pass though is done at the device level
-
more flexible due to independent kernel
-
more overhead
I have been run Docker container in both LXC and VM for a long time without issues or meaningful performance penalties. So I run important single docker containers on top of LXC and everything else in Dockge / Portainer VMs.
I can’t say much to docker in LXC as I’m not using it, I vaguely remember some limitation I’ve read of but if it works fine for you those don’t seem to apply.
A VM has more overhead than an LXC, but with several LXCs maybe a single VM wins on overhead.
I currently have most Docker containers in one VM and am thinking about splitting it, the main reason is that 2 deployments have way larger volumes than the rest. This leads to the snapshots of the VM being very large as well and if I would need to restore from snapshots for a “small” application, it would take super long because of the large ones.
A single VM may be a bit easier on maintenance than several LXCs.
If you don’t have a specific reason to switch, I would not.
