I’m thinking about starting a self hosting setup, and my first thought was to install k8s (k3s probably) and containerise everything.
But I see most people on here seem to recommend virtualizing everything with proxmox.
What are the benefits of using VMs/proxmox over containers/k8s?
Or really I’m more interested in the reverse, are there reasons not to just run everything with k8s as the base layer? Since it’s more relevant to my actual job, I’d lean towards ramping up on k8s unless there’s a compelling reason not to.
Unless you have multiple systems, I don’t think k8s will yield much benefit over plain docker.
The basics can be useful there. The whole idea with k8s is to be able to run applications across multiple hosts in a given fleet. Your cluster can be that fleet! :)
Also k8s is in high demand in the sector, so those are good skills that could be turned into $$
I’m running a 3 pi cluster with k3s at the moment. The main benefit I’ve found is that all my pis run exactly the same software setup as a base so it’s easy to add new ones or replace/update one. I use a deployment management application to push my deployments too which means it’s super easy to redeploy everything if something goes funky.
That can be fun. The benefit of kubernetes is flexibility in the orchestration and (sometimes) scaling. Also the tooling in Kubernetes is more sofisticated compared to plain containers or manual services.
Kubernetes is basically just a finite-state machine that is able to manage a certain number of nodes as a pool of resources. This has added complexity compared to you managing the scheduling (I.e. I install this service on this box and this on this other box), but it also allows for much easier automation.
I think it depends on your scale. If homelab stuff docker is awesome IMO.
I, personally, haven’t done a whole lot of VM work but I do run a metric ass-ton of containers. I can spool up servers in docker compose on absolutely dogshit hardware and have it run serviceably. Also, the immutability of the container OS is really nice for moving things around and/or getting them set up quickly.
Where did you learn so much about Docker? Having a server at home, I’m more inclined to spin up a VM. I would like to learn more about Docker.
Just get started somewhere. I ran traditional VMs for most things before and I would never go back unless it was necessary for something.
Easiest way is just to start using Docker for some service you’re hosting that has a public image available and go from there. If you want a more visual approach there’s stuff like Portainer you can use too.
Also get started early on with docker compose, it makes it much easier to organize your container configs.
VMs are often imperative and can be quite easy and familiar to setup for most people, but can be harder or more time-consuming to reproduce, depending on the type of update or error to be fixed. They have their own kernel and can have window managers and graphical interfaces, and can therefore also be a bit resource heavy.
Containers are declarative and are quite easy to reproduce, but can be harder to setup, as you’ll have to work by trial-and-error from the CLI. They also run on your computers kernel and can be extremely slimmed down.
They are both powerful, depends how you want to maintain and interface with them, how resource efficient you want them to be, and how much you’re willing to learn if necessary.
That sums it up really well.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
That sums it up really well.
I generally tend to try to use containers for everything and only branch out to VMs if it doesn’t work or I need more separation.
This is my general recommendation as containers are easier to set up and in my opinion individual software packages are easier to maintain with things like compose. I have limited time for my self hosted instance and that took away a lot of work, especially when updating.
What I did is install proxmox on the bare metal, setup a vm in which I put the containers.
Proxmox itself stays (almost) completely stock. The only changes I’ve made to it were to add the NUT client package so it could gracefully shut down if my NUT server indicates that the UPS is running out of power during an outage.
In your VMs you can do whatever. Setup OMV, or a stock Ubuntu or Debian vm and install your services on the VM or use Docker/Podman. Setup Fedora CoreOS or IoT vms and host all your services in Podman containers.
The great thing about Proxmox is you can do snapshot backups which take mere moments to complete. Then pass those off to a NAS where they can survive a irreparable loss of your Proxmox server.
You can also spin up new vms as needed to just try to fuck around with new techs or just a new way of setting up your home lab. It gives you a ton of flexibility and makes backing stuff up way easier.
Another great thing you can do is if 3 years down the line you are looking to replace your server hardware with some newer or more powerful stuff you can just add the new device as a node to the cluster. Then you can migrate all your existing VMs over to your new hardware and decommission your old one with very little to no downtime on anything.
The great thing about Proxmox is you can do snapshot backups which take mere moments to complete. Then pass those off to a NAS where they can survive a irreparable loss of your Proxmox server.
Hopefully you put a giant asterix by this point. You need the snapshot AND the original backup. Snapshots are only diffs and can’t survive without their base backup.
This is my exact setup as well. Proxmox with one beefy vm dedicated just to docker and then a few other vms for non docker workloads (eg, home assistant, pihole, jelltfin). I can probably run those in docket as well, but the to worked better as vms when I set them up
Appreciate your take on this and specifically mentioning that you have a VM for Home Assistant. That was a lightbulb moment for me as I like how easy it is to manage updates as an OS install rather than in a Docker container. If I ever get around to rebuilding my server architecture I’m definitely going to do this!
I have a similar setup, but 2 VMs on each of my 2 servers, then on server 1, I have VM A running one test K3s node and VM B running one live (Production) K3s node with the same on server 2, so I can take one server full down for maintenance, but keep my test and live sites running. It’s way overkill, but allows me to learn about how to set up and maintain resilient systems. One day, I’ll do the same for my network :-(
