Docker or Proxmox? Something else entirely?
New server has been acquired. Debian 13 has been installed.
GS308EP switches have been acquired and installed.
Now, I'm working to migrate to the new machine. 3 1/2 years ago when I started futzing with Docker, I sorta followed guides and guessed, abused it trying to make it do things it wasn't designed for, and flipped switches I likely shouldn't have flipped, so the set up is more than a little shabby.
As a result, I'll likely end more redeploying than migrating the containers.
So rather than go forward with Docker blindly, I want to reassess whether I shouldn't look into Proxmox, LXC, or Podman instead of Docker, or maybe something else entirely?
Work is just about done dumping ESX for Nutanix, but both of those seem overkill for my needs.
Of course the forums for any of the solutions make their own out to be the best thing since sliced bread and the others useless, so I'm hoping to get a more nuanced answer here.
Vendetta9076
in reply to JeanValjean • • •like this
TVA likes this.
SpikesOtherDog
in reply to Vendetta9076 • • •abeorch
in reply to Vendetta9076 • •Selfhosted reshared this.
uncloaked
in reply to abeorch • • •abeorch likes this.
Scrubbles
in reply to abeorch • • •felbane
in reply to abeorch • • •It's not worth the headache IMO. Just run a docker VM and use lxc for the one-off systems that you want to experiment with.
I have a "production" docker VM and a "sandbox" docker VM and prod only ever runs compose files that I've vetted in sandbox. Super stable, basically bulletproof, and still has the flexibility to experiment and break stuff without affecting my core services.
iturnedintoanewt
in reply to Vendetta9076 • • •Imacat
in reply to Vendetta9076 • • •Blastboom Strice
in reply to JeanValjean • • •Creat
in reply to Blastboom Strice • • •Especially without any additional context or knowledge about their background, directed at someone clearly only starting out, this is incredibly bad advice.
Edit: typos (italic), sorry that was probably hard to read.
Blastboom Strice
in reply to Creat • • •Oh I didnt see the downvotes😅
Hm, to explain my reasoning:
I see that in the end they ask for a nuanced answer, rip, I failed to do that
ryanpdg1
in reply to Blastboom Strice • • •Hahaha. I see you're being down arrowed a bit... But I was thinking the same thing.
Reason I was thinking nixos was because op was talking about " flipping switches" they couldn't flip back.
Nixos has that sorted out.
Plus... It sounds like op isn't afraid of trying new and possibly difficult things on a new server they've acquired
The truth is that proxmox is likely the best option here. But nixos could be a fun ( type2 fun) challenge if they're into that kind of thing
glizzyguzzler
in reply to JeanValjean • • •If you’ve got Debian already installed, I cannot resist advocating for Incus (stable branch from Zabbly repo with web ui blog.simos.info/how-to-install…) in lieu of proxmox. Does the same thing but you don’t have to rip out the kernel Debian uses.
With Debian 13 you have access to podman quadlets, use that for any non-vm needs. The ease of docker compose files easily removes reason for programs in LXC containers, and podman removes reason for docker in an LXC. LXC is left only for programs that aren’t containerized. VMs for security DMZ. Podman for bulk of stuff you want.
Good luck!
How to install and setup the Incus Web UI
Simos Xenitellis (Mi blog lah!)like this
arjache likes this.
tofubl
in reply to glizzyguzzler • • •like this
TVA likes this.
koala
in reply to tofubl • • •Came in here to mention Incus if no one had.
I love it. I have three "home production" servers running Proxmox, but mostly because Proxmox is one of very few LTS/comercially-supported ways to run Linux in a supported way with root (and everything else on ZFS). And while its web UI is still a bit clunky in places, it comes in handy some times.
However, Incus automation is just... superior.
incus launch --vm images:debian/13 foo
, wait a few seconds thenincus exec foo -- bash
and I'm root on a console of a ready-to-go Debian VM. Without--vm
, it's a lightweight LXC container. And Ansible supports running commands throughincus exec
, so you can provision stuff WITHOUT BOTHERING TO SET UP ANYTHING.AND, it works remotely without fuss, so I can set up an Incus remote on a beefy server and spawn VMs nearly transparently. +
incus file pull|push
to transfer files.I'm kinda pondering scripting removal of the Proxmox bits from a Proxmox install, so that I just keep their ZFS support and run Incus on top.
SayCyberOnceMore
in reply to koala • • •Hmm, I setup a Proxmox machine a while back because, well, all the cool kids seemed to do it - and plenty of "support" on youtube
I found Incus and it just seemed better, but it was harder to find info on (back then) and seemed a little unready
Now, I regret not sticking with my gut instinct as I've got to basically rip out Proxmox to get Incus in, which means all my VMs are prisoners (and us: 1 VM is Home Assistant!)
So, do you know if it's possible to migrate my VMs across to Incus, or is it literally wipe drive, start again?
(Obviously the data in each VM can be backed up & restored into new VMs)
koala
in reply to SayCyberOnceMore • • •I haven't tested this, but I would expect there to be ways to do it, esp for VMs if they are not LXC containers.
(I try to automate provisioning as much as possible, so I don't do this kind of stuff often.)
The Incus forum is not huge, but it's friendly, and the authors are quite active.
glizzyguzzler
in reply to tofubl • • •I’m looking at Opnsense on an Incus VM soon, what was your fight there? Good to know what I’ll hit ;)
Agreed on that path - some networking (like mimicking proxmox’s bridge connections which give VMs their own MAC/IP) takes effort to find the solution. But the basic LXC/VM-shares-your-IP works super easily and the script ability is great. Plus it doesn’t feel like a yoke on your system that is heavy and drives it, but just another application! I feel it’s close enough, and when you get it where you want it, it’s perf. I assume they’ll get “one click” solutions for the harder stuff baked in as they get more attention and traction.
tofubl
in reply to glizzyguzzler • • •"Just" some highly specific VM settings, in the end. I don't know much about that, and terms like qemu don't mean anything to me so I followed blog posts until it worked. (This one and maybe this one, I think.) It's possible that it is actually trivial.
It's been a while, but I can look up what I have when you need it. Feel free to ping me!
Yes, it was exactly that: Once I got the NICs set up the way I wanted them it was a breeze and everything just works. And I really like that I made every part work myself, no magic. I learned a lot, and wouldn't have had I relied on Proxmox fiddling with the right parts for me.
Run FreeBSD 13.1 / OPNsense 22.7 / pfSense 2.7.0 (and newer?) under LXD VM
Linux Containers Forumglizzyguzzler
in reply to tofubl • • •beerclue
in reply to JeanValjean • • •frongt
in reply to beerclue • • •like this
TVA likes this.
reluctant_squidd
Unknown parent • • •I concur. Podman is superior in my opinion. It’s more secure by default (rootless containers) and can do pretty much everything docker can do naively (you can literally alias docker to podman in your shell and it will work)
It’s not as easy to find info on some of the systemd specific stuff (Quadlets), but once you figure that out, it’s pretty amazing.
I ended up making up my own scripts to allow me to create new system users, pre-loaded with aliases and shortcut functions to make my life easier ( automatic quadlet container file generation, pre-set network rules, etc), but it is not required.
All the info is there, but starting out it can be a bit overwhelming.
My containers are pretty much self sufficient now. I just intervene when something needs major updating or config changes
chocrates
in reply to JeanValjean • • •Proxmox is a hypervisor. I would install that as your os if that is something you want.
If you just need containers, podman is good. Docker is fine, but podman is open source.
That being said, I run proxmox on my server and have various vms running. I have a laptop that I will use to play with containers and if it turns into a service I want I'll make it run on proxmox, either as a container itself or as something running on a VM on the hypervisor
like this
TVA likes this.
hobbsc
in reply to chocrates • • •Possibly linux
in reply to chocrates • • •Creat
in reply to JeanValjean • • •Proxmox and Docker don't really do the same thing. They live in the same area, but the coverage is very different. You can always use docker when your host is running proxmox: either individually or in groups inside of an lxc, or all in w dedicated VM, or even natively on the same house if you prefer chaos. But you can't do the opposite: Sometimes you just need a VM. Maybe you only need a couple of devices, and you know they run on or are even designed for docker, then that's the better option. In all other cases, and when just getting started, proxmox is just the way more universal solution if you're only planning on having a single host (for now).
The management tools in proxmox are great. The community scripts are a fantastic resource and only work with proxmox. I would suggest you set it up natively, not on top of Debian though, even if that's already installed. Not the least of the reasons are to be able to use ZFS easily, including on on the boot partition (select that in the installer).
Finally, if you're gonna stick with docker, like others said: consider podman. That really does the same thing docker does, but it's fully open source. Arguably it's better in some areas, but on the flip side might, in occasion, require fiddling with something intended specifically for docker and using advanced setups.
Also there really is no wrong answer, either. And you can always change whatever you choose.
like this
TVA likes this.
abeorch
in reply to Creat • •TVA likes this.
Selfhosted reshared this.
Creat
in reply to abeorch • • •TVA likes this.
Encrypt-Keeper
in reply to Creat • • •Encrypt-Keeper
in reply to abeorch • • •Possibly linux
in reply to Creat • • •Docker community edition is fully foss
I like Podman but In don't think foss is the driving factor in this case
like this
TVA likes this.
Lka1988
in reply to JeanValjean • • •I run Proxmox in my small cluster. Proxmox runs multiple VMs that each run various groups of docker containers.
They're on different levels.
like this
TVA likes this.
tazeycrazy
in reply to Lka1988 • • •frongt
in reply to JeanValjean • • •Proxmox can run lxc containers natively.
Personally I keep a Debian VM for docker, a holdover from before hypervisors supported containers natively. I use docker compose and it Just Works™.
tazeycrazy
in reply to frongt • • •warmaster
in reply to JeanValjean • • •like this
TVA likes this.
Possibly linux
in reply to warmaster • • •like this
TVA likes this.
metaStatic
in reply to Possibly linux • • •Possibly linux
in reply to metaStatic • • •metaStatic
in reply to Possibly linux • • •abeorch
in reply to JeanValjean • •I am not in a position to judge but Id be interested to hear comments on this position.
From the Proxmox Discussion Forum
Linux Containers (= LXCs) and docker containers are something completely different. LXCs are system containers. These contain a full OS except for the kernel which is shared with the host. You will have to administrate them like you would do it with a VM.
Docker containers are application containers where you containerize a single appliance. You don't individualize/upgrade them. You just throw them away and create a new one. So you are quite limited what you can do with them, as all you can configure is what the container creator wants you to be able to change.
So if you want a service as secure and independent as possible, or if you want to use Win/Mac/FreeBSD, use a VM.
If you want something like a VM running Linux, but you want less overhead, at the cost of security, use a LXC.
If you just want to run some services but you don't want to invest time on learning how they work or administrating them, then running a VM with docker would be a good choice.
like this
TVA likes this.
Selfhosted reshared this.
ruuster13
in reply to JeanValjean • • •like this
TVA and metaStatic like this.
jimmy90
in reply to JeanValjean • • •ducklingone
in reply to JeanValjean • • •punkcoder
in reply to ducklingone • • •WhyJiffie
in reply to ducklingone • • •OminousOrange
in reply to WhyJiffie • • •ducklingone
in reply to WhyJiffie • • •Zanathos
in reply to JeanValjean • • •Don't go Podman. When I started years ago I installed Fedora with the "containerization" option. This installs podman, not docker as I'm sure most know. I did not.
Podman works great for the most part, but it's slight differences from docker will have you fighting tooth and nail for certain services to work correctly. And not many (if any at all) have any documentation on getting their containers working with Podman of they don't start. If you make a GitHub issue asking why or how to get things running in Podman because their docker stack doesn't work flawlessly like it will in docker, good luck getting help (Mailcow comes to mind specifically here).
Looking back, this decision really shoehorned some very fundamental ideals about containers in my mind, but it was a long fought road I would not choose again. The knowledge I gained about containers with docker would have come soon enough on the easy road.
And yes, you can install Docker on Fedora, but I was much too far down the Podman track before finding out. My environment has changed drastically as of late and most things have been migrated to docker apps in Truenas now, living directly next to their storage as intended (the arr stacks really take a performance hit running their databases over NFS once you have a lot of media for example).
Quick note about Proxmox after coming from ESX myself - it sucks compared to ESX. I've tried to move away from it and Nutanix was the closest I could find to ESX, but after my server started complaining it's drives were not compatible I jumped ship to avoid any write damage to them. I'm downsizing my lab now, I have proxmox running in 3 small NUCs with CEPH storage share and it's working pretty good. Would love to run ESX or Nutanix instead, but they require a loaf of bread in resource requirements where proxmox only needs a slice of bread in comparison.
Infernal_pizza
in reply to Zanathos • • •Zanathos
in reply to Infernal_pizza • • •I was a little unfair in my post towards Proxmox. It really is a great solution and I can't really complain, but it sucks in comparison to ESX where many "custom" items are still hidden in the cli or custom configuration items,. Many of these things are available in the GUI in ESX which is a pretty rough translation for some that have worked in ESX for many years like myself. ESX isn't without it's CLI moments but they are rarely ever needed, and if needed only for drastic measures.
The UI is not very intuitive and really looks quite dated too. ESX, Nutanix and XCP-NG have much better interfaces imo, and if Proxmox could throw some of that extra money they've earned from the VMware exodus in their UI it would be worthwhile.
Again, I shouldn't complain but as I get older there's not much "tinkering" time anymore, and the less time I have to sift through forum posts or official documentation on why something isn't working as intended, the more easily frustrated I get.
Evotech
in reply to JeanValjean • • •ghurab
in reply to JeanValjean • • •prettybunnys
in reply to ghurab • • •Miggi
in reply to JeanValjean • • •Maybe use Docker with a UI like
lazydocker
orPortainer
.If you feel safe with Docker, perhaps experiment with Proxmox as a base. Then you can separate your environments into different VMs.
WhyJiffie
in reply to JeanValjean • • •docker in a vm in proxmox!
no but really. I don't like that the vm host's kernel is shared with all the containers.
then maybe have 2 or 3 VMs, one of which hosts essential services (not like your nextcloud, but services that are necessary to make the (internal or whole home) network work like DNS and DHCP, maybe monitoring, and another that hosts your convenience services (jellyfin, nextcloud). the first is easier to fix if it breaks, the second is not the end if the world if it does. maybe also have a third for services that would be painful to have offline (nextcloud), but try to limit it to few tasks because the goal is simplicity. this setup can help with distro upgrade difficulties, and vm custom configuration issues when you have set up something that seems to work but will prevent VM startup sometime in the future.
podman could be a good idea too.
Appoxo
in reply to JeanValjean • • •maarvin
in reply to JeanValjean • • •Proxmox with all of its warts appears to be the better hypervisor. If your host has the headroom like others suggested: proxmox with a VM hosting docker/podman.
I have to say running proxmox saved me from running to the basement or opening up IPMI a few times already when making suspect changes to a VM that otherwise would have taken down the network of my host.
ATM I’m running Proxmox with Nixos VMs running mixed docker/podman containers. It works out pretty well for my use case and with some opentofu fiddling I have most of my infrastructure defined in config files if that’s a rabbit hole you want to go down.
As for my experience with docker vs podman I have to say podman can pretty much do whatever docker does with the exception of docker swarm. You may have to do some digging to handle more advanced networking/gpu setups.
The only thing I have running docker atm is a gluetun container because container to container networking took more than 15m of research with podman so I fell back to the very well documented path using docker.