Hacker News new | past | comments | ask | show | jobs | submit login
Transitioning from Docker to Podman (redhat.com)
463 points by indentit 3 days ago. | hide | past | favorite | 261 comments

It's compatible with cgroups v2 unlike the standard Docker. If you're using Fedora, you have to add a kernel parameter to Grub to use cgroups v1 instead.

RedHat seems to be pushing a standard ecosystem for Linux: systemd, Wayland, SELinux, GNOME, and now maybe podman. I've been on Linux for a while; it's a welcome change from all the fragmentation I'm used to. Whereas others try to work around the kernel and implement their own things in parallel (see: Canonical's AppArmor, LXC, OpenZFS), RedHat just goes with what Linux already has like SELinux/cgroups v2/btrfs, which I think is more likely to last and just feels better. If RedHat goes away, I'm fine since I'm ultimately only relying on Linux features. If Canonical goes away, then I'd have to switch to a different stack. That's probably why government, enterprises, Amazon, etc. still prefer RedHat.

What you are noticing is the RedHat “upstream first” policy, and it’s very practical. They try to minimize RedHat specific modifications and instead contribute improvements directly to the original projects.

This reduces the amount of code they are responsible for maintaining, and everyone involved gets the benefit of open source collaboration.

They have a blog post about it here:


> RedHat just goes with what Linux already has

More like, Redhat/IBM defines what Linux has simply by paying most Linux devs, and of course they take advantage of it in userspace and sys management.

cgroups/namespaces, systemd, gnome (and it's integration with systemd) plus others are two-sided swords: one one hand they create new functionality, but on the other hand the price to pay is concentrating Linux know-how in one place, and greatly diminishing portability of Linux apps vs other Unix O/Ses and even other Linuxes that don't want to go with the program of absorbing ever more functionality into kernels and system frameworks for no real reason other than monopolization.

Unix was designed as a simple portable operating system in very short time. With 30 years of development, the situation today could also be interpreted in such a way that Linux devs just can't stop to add code. Time will tell if Linux can be maintained if the original generation of devs step down. I'd feel more comfortable if we've let kernels stay minimal rather than becoming kitchen sinks. Apart from better portability, this would've also enabled younger devs to start from scratch rather than having to maintain daddy-o's monstrosity.

>and greatly diminishing portability of Linux apps vs other Unix O/Ses

Thank's god for that! An OS should build on its strenghts, not be a least common demoninator for portabillity's sake. If that was ok we might just as well just use one OS

Indeed, for me the best UNIX experiences that have been created so far have been NeWS and NEXTSTEP (macOS), and I wouldn't mind if Linux would just standardize on one stack as well.

Having used several UNIX flavours since 1992, I can't relate to what is simple in UNIX, unless you are speaking about V6.

As a 23 y/old younger (well, young-ish) dev, it strikes me that this isn't anything new. Old commercial solutions since nearly the beginning have been predicated on proprietary solutions specific to some vendor's UNIX based OS. HP-UX, AIX, even back to the really early commercial releases of v7 all seem to have a common theme of having that unix core but with a lot of nice features added on, typically with pretty insane and convoluted frameworks. Tru64's clustering, AIX's management interface, all the competing UIs and management interfaces. All nice features deeply integrated into a proprietary solution as a form of vendor lock-in.

To your point, it seems like eventually this ends poorly for the incumbents, when they become too bloated and too difficult to extend or modify, since you have to work around some over-engineered high-availability process clustering to touch anything.

In Linux's case, I think the kernel itself is actually fine, but the user land is increasingly tightly glued to an increasingly convoluted set of RedHat stacks.

I mean, it always has -- RedHat made the modern posix threads library, Linux's PAM, and several other core parts, but at least those were pretty modular and "unixy", by comparison to the monolith being developed now.

Anecdotally, I've worked with developers at Redhat and Canonical, and the Redhat developers had passion. They really believed in open source and the linux community. In comparison Canonical seemed like Just Another Software Company to me.

My experience has consistently been the opposite.

With Redhat, Canonical, or both?

This sounds anecdotal to me.

As anecdotal as the person he is replying to.

I mean, yeah, thats what I said right?

Yeah so? Parent (and grandparent) talked about their direct experience. That's by definition anecdotal.


I think governments prefer redhat because they are consistent and they accept gobs of cash to promise security patches to software that are past their support window.

Governments also prefer Redhat because they early on and continuing today have shown a commitment to the market by investing in getting their products certified to meet government required standards including developing the required capabilities when not already available in the Linux ecosystem, developing documentation and software to assist deploying systems in a manner consistent with government security requirements and participating in conferences, working groups, interoperability tests and other government targeted or sponsored forums.

Some other Linux vendors now also do so but started much later and even now do so to a much lesser degrees.

We have our LTSS -> Long term support at SUSE. Just in case you're a big government that wants to throw us lots of money.

Off-topic, but what’s you experience working at Suse? I know a former employee, and what he told me sounded hellish.

I'm an Ex-SUSE employee, and I quite enjoyed it - since I left a few years ago, I don't think there's been any turnover in my group since.

I know two people who work or recently used to work at Suse and they liked it quite a bit.

I love it. Not that it doesn't have it's issues. I've been here 10 years now though.

Pretty much this, they know how the game is played and they play it.

I don't know about the gobbles of cash, but I'm pretty sure Canonical supports Ubuntu releases way past EOL for paying customers - they call it the ESM (Extended Security Maintenance), which 'provides important security fixes for the kernel and the most essential user space packages in Ubuntu'.

Depends on your definition of support I suppose. Redhat employs so many more low level engineers they can generally get you a z stream patch or something to fix a legitimate bug. Canonical is an exceptional marketing company, but I've found their engineering to be lacking when it comes to enterprise. As someone who's worked as a linux monkey for companies who pay Canonical for support and for companies who pay Redhat for support, I've always found the Redhat support to actually solve the root cause. When we filed what we thought were legitimate bugs with Canonical, the common response was just to "upgrade to the latest version of Ubuntu". When we asked them to confirm that doing so actually fixed our problems we just got crickets.

It tends to also be in the opposite direction, big expensive enterprise software is supported on redhat forever and ever. And the issue isn’t just security updates, but full software and hardware support for the same stack for many years.

Lots of people don’t want to recreate their stack every few years for the latest cool thing.

Ubuntu ESM is 8 years, significantly shorter than RedHat's 12 year maximum.

FWIW, Docker support for cgroups v2 is coming in 20.10 AFAIK

btrfs is depreciated in redhat distribution

They just made it the default in fedora. It is likely not deprecated for unreleased RHEL versions

I just installed Fedora yesterday and was pretty confused at that part, after reading many articles and posts about RH dropping btrfs.

Fedora is a community distribution despite what many think. It is not controlled by RH in any way.

"Fedora is developed and sponsored by Fedora Project and Red Hat."


Both of you are right. Red Hat pays for Fedora in a lot of ways, but does not control it. There are obviously a lot of personal and professional relationships that gives Red Hat influence over Fedora, but the purse strings are never used to control it.


On the other hand isn’t btrfs becoming the default in Fedora 33, thus eventually making a comeback in RHEL?

Yes that may well happen.

It's also noteworthy that the BTRFS change was driven by community members, mainly from Facebook.

We use it as the default at SUSE. In SLES and openSUSE.

If you want to use btrfs on RHEL8, you still can: go to https://elrepo.org/ and install elrepo-release, then install kernel-ml from their -kernel repo and btrfs-progs from their -testing repo.

... at which point you lose the support you're paying for from Red Hat, thus defeating the purpose of using RHEL in the first place.

So if you want to use Btrfs, you might as well use it on something besides RHEL.

Running an unsupported kernel doesn't mean that support won't help you at all. It just means they won't help you with kernel-related issues. There's still a lot of value in support for the rest of the system.

I have heard that they will still try to help you, they just won't guarantee it works on btrfs.

Red Hat deprecated BTRFS in its products (and does not have anything like ZFS).

Its focuses on XFS from 1993.

Its Stratis idea is just a joke comparing to what ZFS is.

They should finally admin (like Canonical) that OpenZFS is the way of future filesystem for now.

Linux world should put their religion (GPL) aside here.

Besides above filesystem 'issue' the PulseAudio, systemd and SELinux are not hated without reason ... many people just 'love' what Lenny has to offer.

At my company, we run our CI/CD (Jenkins) using the Docker-in-Docker paradigm to facilitate easy maintainability of the CI itself and allow us to run containerized builds.

When we shifted to RHEL 8, we attempted to move this over to Podman and it went miserably (this was back in November 2019). The main reason being is that podman-in-podman doesn't work and had bugs (at least back in Nov 2019). Maybe it fixed now but this was our experience. We ended up doing quite a bit of analysis on podman only to conclude it's simply not there yet relative to docker (ecosystem and ergonomics).

There are quite a few corner cases that docker quite simply supports out of the box beautifully that podman doesn't support or just has bugs.

I like the what the project is trying to solve by being daemonless, but this is not as simple as a drop in replacement for docker that RedHat markets it as (alias docker=podman).

We ended up sticking to docker professionally and personally, I am still using docker over podman. The ecosystem and ergonomics are just far too nice to give up over podman.

Maybe you are just too far into docker. I noticed that a lot of default workflows (needlessly) depended on docker running with privileges. One big reason for that seem to be Mac users that only know docker from inside a VM. However, if you think about what you're really needing for CI you will easily see that docker-in-docker gains you nothing. You can as well use plain docker (or podman). The same holds for privileges. No CI operation should need privileges, if only for the reason that it should never alter the CI system itself.

I encourage you to not take the standard workflows as a given and really think about what you need and I bet you either end up with a use case that can be covered by rootless podman or something that requires real VMs anyways.

Okay, I want to build a container image using gitlab CI, which runs builds in docker. How would you like me to build an image without using docker in docker, or buildah in docker?

We use kaniko[1] in Gitlab CI and it’s working great for us. It’s annoying the kaniko image requires us to specify the entrypoint. There’s some peculiarities with Dive [2], but otherwise it’s been a very easy migration.

[1] https://github.com/GoogleContainerTools/kaniko

[2] https://github.com/wagoodman/dive/issues/318

I tried kaniko and didn't like it.

Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler. I know there's `--no-push`, but that's a poor substitute for independent `build`, `tag`, `push` build steps IMO.

Do you have any way of running / debugging locally with GitLab CI plus kaniko? Can you run your build pipeline locally on your workstation against uncommitted code?

IMO I can build a way better local workflow that allows me to run builds _before_ committing with Drone (`drone exec`). I can toggle between a locally bound Docker daemon or a DIND environment that's going to be virtually identical to the DIND environment on a build runner. The `push` step doesn't run locally plus the secrets needed to push are only accessible from an official runner. I can run it on Windows or Linux (and likely Mac) too.

I've been trying to find a self-hosted CI system that's really good at building Docker (or OCI) images and I don't think any exist. They all have short-comings. Having build, tag, push act like an atomic build step is one of the areas where I think most fail. So many claim to enable repeatable builds but none actually do AFAIK. Whoever writes the Dockerfile needs to know a lot about how images are built to have the slightest chance at creating a repeatable build. A great example is having `apt-get update` in a Dockerfile. That command _always_ returns zero, so by itself it makes builds non-repeatable.

Sometimes I have a tough time reconciling the development industry because things just don't make sense to me. I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.

People used to complain about seconds because the wait was "too slow" for good local iteration, but now waiting for minutes is a "good" build system. Seriously WTF?

I guess I got on a bit of a rant...

> I remember people complaining about Gradle start times so much they came up with the Gradle daemon. Now no one bats an eye at CI based build systems where you have to commit your code, wait for a runner to get provisioned, wait for Docker or the OCI runtime to spin up, and wait for your project to actually build on some anemic VM.

I want this framed or sewn onto a pillow or something.

It's amazing what we can build, it's baffling what we have built.

> Having build and push as part of the same job is frustrating and I view it as a sign that a CI system is built with the expectation of having everything happen post commit by shoveling money into a CI auto-scaler

No, the idea is to build and push images you can test directly afterwards in the same conditions. With cache and such, build times shouldn't be too long

Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally.

If you have long CI times, that can hinder development productivity and should be improved as much as possible, or a local replica of the CI needs to be created.

> Gitlab CI just runs shell commands, it's pretty trivial to pull the same image its using in the job, and run the same commands locally.

In fact, they already provide a utility for doing this: gitlab-runner exec[0].

[0]: https://docs.gitlab.com/runner/commands/#gitlab-runner-exec

It is really working for you? Presumably you setup kaniko to build the images first that then run the second part of your pipeline. To run the images, you need to tag them with something that you use inside the gitlab-ci.yml, right?

Now what happens when two people push code that make changes to the containers at the same time?

GitLab CI exposes the pipeline ID and it would work for that.

kaniko is really really slow:

    - https://github.com/GoogleContainerTools/kaniko/issues/1392
    - https://github.com/GoogleContainerTools/kaniko/issues/970
    - https://github.com/GoogleContainerTools/kaniko/issues/875
I've observed basically the same stuff.

Your problem is the assumption that a gitlab CI process requires a docker image. If you do the following, you are fine:

* Switch to a shell runner

* Put the CI dockerfile into your repo

* Provide an entry script for CI that builds the container on-demand (and manages caching/cleanup) and then runs the tests/whatever inside that container

The point here is that docker/podman provide you with everything you need as long as you have full control. By using gitlab's default CI, you relinquish this control.

But how would you run integration tests between multiple docker containers? We launch our services in these docker containers and do some integration / e2e tests, and this very much requires Gitlab CI to launch docker containers while inside a Docker container.

After using dind for some time we chose to just mount /var/run/docker.sock and keep using the host machine’s docker instance (mostly for the cache), but all in all dind was working fairly well.

To be honest, to say “you shouldn’t be doing that” is missing the point; one should be able to do anything they want. In my opinion, the root cause here is docker’s client/server model, which is fixed by other container runtimes such as podman and rkt (which unfortunately is deprecated). One should be able to just launch containers as if they were just another process.

To the contrary. You should never expect to be able to do what you want with a system provided by someone else. Gitlab has its design decisions in their default setup and you need to take control to cater your use case. This herd mentality of "everyone is doing it this way" is really fundamentally problematic.

Nowhere in my post am I saying "everyone is doing it this way".

You started out with asserting "you're too far into Docker", we bring up valid use cases for docker-in-docker, and then you saying "This herd mentality [..] is really fundamentally problematic" is really not adding a lot to the discussion.

No one brought up valid use cases for docker-in-docker. They brought up the issue that gitlab mandates docker as an interface (which I totally understand, btw.).

For instance the "how do I compose multiple docker containers" is trivial when you can just execute a script that runs docker or podman. If you really want, you can use docker-compose.

AFAIU you can't switch to a shell runner unless you self host gitlab runner.

That is exactly my point. Gitlab uses docker runners because it is much simpler for them. But why should you be constrained by what's simpler for gitlab?

Well, it's free to use GitLab's hosted runners. You have to pay for your own build server if you want to host your own...

Kaniko is a choice: https://github.com/GoogleContainerTools/kaniko

Userland docker builds

https://github.com/GoogleContainerTools/kaniko can do it but I just use podman / buildah with storage driver as vfs, and then neither requires root / high privileges.

I'm not sure why it doesn't do this by default. Performance I guess.

Our org just adopted Makisu[1] which works beautifully in our GitLab CI pipelines. After a week of setup and fiddling with the settings, we were able to migrate all our build jobs to Makisu and haven’t looked back. Build times are great too, especially if you set it up with a Redis cache.

[1]: https://github.com/uber/makisu

Have you used / considered kaniko before? If so, what are the advantages of Makisu?

Why not simply use that then?

We're using podman in containers inside gitlab ci.

We're _also_ running tests of containers inside containers in containers using gitlab-ci.

The main workaround we've applied is using crun as the runtime rather than runc.

Kaniko :) no DIND weirdness or privileged containers necessary

Lots of folks recommending kaniko, which we use. But it seems quite slow compared to docker build. Is this not an issue for others?

A container image is just an archive file with some metadata, could you take the same approach as Google Jib?

Why do you say that docker-in-docker buys him nothing? It's not obvious at all and you go into no detail whatsoever to back up your opinion.

In my experience, that is not true at all. Docker-in-docker allows me to deliver smaller images that can fit into a CI flow as language plugins instead of shipping a beastly 5G docker image with every possible language runtime I need to support for my CI tool.

It is because to build the image using docker requires the docker client to talk with a dockerd daemon, so one has to configure the client to access the dockerd which allow untrusted code to run as root in the host.

Docker-in-docker is a workaround to make docker work in CI.

Basically a security nightmare and bad design that podman doesn't have.

Any build script can do serious damage to the environment it runs in. Before docker, you'd have to create a new VM from time to time because the build agent had rotted away or died in an altercation with a bad build.

Docker in Docker in CI is like a lock on a door. It keeps honest people from being naughty, and is fairly efficient about it.

I don't think the question is "should I run CI in docker in docker," it's whose CI should I run in docker in docker. Me in my coworkers can share docker images. Customers or freeloaders cannot. So if that's in your problem domain, then you're right, it's a bad idea. But it isn't for most people.

You do know that spinning up a new VM only takes a few seconds? With projects like https://firecracker-microvm.github.io/, the difference between launching a new Docker container or a new VM is negligible.

This works great if you own or rent the hardware, but most cloud providers don't allow nested virtualization.

The cost is not spinning up the vm, it’s maintaining the images. Docker composability reduces the combinatorics problem to a dull roar, and democratizes some of the maintenance effort. You want an image with the bug fix from the latest point release of python? And you need it by noon? Knock yourself out.

Although there are tools to convert docker images to vm images. I expect if I were running community CI infrastructure, getting really familiar with those would be high on my priority list.

The other option that works really well in a single user environment is to bind to the runner's Docker daemon. That way builds run as siblings of the runner's daemon rather than as children via docker-in-docker.

The huge issue with that is security which is why it's only really practical for a single user or a small group of trusted users. A secondary issue is that (I think) builds can't run simultaneously because they can trample each other when tagging images (since all images are on the runner's daemon).

If I had to build a Docker focused CI system I'd think about using Weave Ignite (AWS Firecracker) to spin up VMs for runners with the Docker socket bound like described above. That way you get all the convenience of binding the Docker socket, but the isolation of a VM that gets thrown away after the build step (or pipeline) finishes. That idea also fits well with local running / debugging IMO because you can bind to the Docker socket on your development workstation (assuming you're not running a large build of parallel tasks which might be an unrealistic assumption).

For us it’s a matter of the CI tool fetching the source code for the docker image, then running docker build, and not necessarily immediately. So you have ‘docker build’ happening toward the end of a set of other tasks. Which I’d really like to have running on a fresh VM or container.

You could separate those into two builds, but the reason they are together is so people think about deployment, and in case any structural changes to the code need to coincide with deployment changes. For instance, breaking changes in APIs. I need a new version of tool/library and I need to change how I call it.

Kubernetes as a layer of indirection is another solution.

Kubernetes is a consistent management api for linux (so then you don't need to interact with iptables, mount and all that "hard stuff").

I don't think kubernetes is a solution for context of building an image (a rootfs tree into a .tar.gz file).

Unless you are using kaniko which extends the kubernetes api to add the capability of creating images, but that is handled by kaniko itself via the same api.

I was probably unclear. Kubernetes is a good solution for managing containers (obviously). I use it for CI and it works very, very well, though the CI tools still have more features they could add with the integration.

> beastly 5G docker image

my beastly 12GB image that even includes Matlab wants a word with you

>> beastly 5G docker image > my beastly 12GB image that even includes Matlab wants a word with you

Perhaps in the next 10 years we will be rediscovering packages. :P

If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s

> If you are in the business of charging complex prices per bits over the network, then docker seems to be quite a good investment and making it as popular as possible is a good strategy to print money. /s

True, that.

To be fair, at least it allows me to avoid lots of the brokenness of Python packaging.

Is always good to report packaging bugs so then people can fix them, do you have examples of python packages that can be improved?

See my previous comments.

tl;dr pip silently breaks my environments, mostly connected to upgrading numpy and other scientific/data science libraries.

Well, to be fair, it is packages - I'm just using Docker (for this section of our stack) as a different sort of VM, essentially. It runs a service manager and a VNC X session, for chrissakes ;)

Our images are 35GB, and I've spent much of the last two weeks breaking up files so we don't hit the 8GB per file limit, and my next week will be trying to avoid hitting the per-layer limit.

I think the ideal is using Nix to manage development dependencies and to handle building minimal docker images for deployment.

Why not just use nix at that point? You can at least retain the advantage of having truly immutable and reproducible builds.

I use docker for most of my clients' work but for in house stuff I just use nix.

Docker and Nix are mostly orthogonal technologies though. Nix is an excellent build tool while Docker is useful as an universal software distribution format. They're really useful together because Nix is actually good at creating compact Docker images. But of course, if you don't have to worry much about distribution, it would be much nicer to stick with Nix as you've mentioned.

Well, just nix works pretty well unless you want to deploy to K8s or Fargate or similar.

>One big reason for that seem to be Mac users that only know docker from inside a VM

That has not been the case for a good while now... Docker has been running directly on a hypervisor on the Mac.

Honest question, what's the difference between "from inside a VM" and "directly on a hypervisor"? I always thought it meant the same thing.

Docker for Mac is still running a VM via LinuxKit/HyperKit, but there's a big difference in overhead between LinuxKit and running VMWare/VirtualBox.

Well, in the case of Mac, basically performance, as you still need linux libs, kernel API and everything. But it's much lighter-weight than something like Virtualbox, VMWare, etc.

In Linux iirc, a hypervisor can share such resources with the host system (since they are both Linuxes).

I'm not married to docker either, and love the idea of podman. In my case the blocker for migrating was related to docker-compose.

podman-compose doesn't support as much as I needed for my current deployments - although offhand I cannot recall what was missing.

I'm looking forward to trying things again in a few months, but those corner-cases can be real pains to deal with.

We're working hard on docker-compose support. With the new Podman 2.0 REST API with a Docker compatibility layer, we are close. Not quite there, but definitely strategically on the roadmap.

> Docker-in-Docker paradigm

Apologies for the patronising comment, but do you really mean that? Docker in Docker works but is intended for the developers of Docker to debug Docker itself. Usually for running Docker from within a container, you just hook up the Docker client to the TCP port of the Docker daemon running outside the container, which isn't strictly Docker-in-Docker.

I ask this in case you're trying a wildly use case (if you really are running true Docker-in-Docker), or are making an unfair comparison (if you're just using the usual Docker client in container to daemon on host). In the latter case, I must admit I don't know what the idiomatic alternative would be for podman, given that I know nothing about it except that it's daemonless (and even that I only learned by reading your comment).

I think what most people mean when they say docker-in-docker is in layman terms. You simply just a way to do "docker run" from within another container. Whether that's done by actually running docking daemon inside or mounting something from the host or whatever else i don't really care. I just want it to work.

Sure, there are some details you might want to control, like image caches and such being shared with host, i just find there is a lack of documentation and best practices of how to do nested docker, if that is even intended to work or if mounting docker.sock is an unsupported hack. Most information found about this is scattered on shady blogs.

For the examples of CI brought up the use cases are fairly obvious, you have a jenkins installation with x plugins installed - running as a container. Within this jenkins you are building multiple different projects which all require their own respective image to build. As a project developer here i don't even want to know if this jenkins is a bare metal, a vm or a container. Here docker is more used to bundle all the dependencies, not for strict security with perfect containerization.

Docker in docker is a very common pattern in CI, in some cases is the only way to achieve certain tasks.

There is also Kaniko which works quite well:


What kinds of tasks require two levels of containers?

Your CI server running in a container that is itself executing Docker commands.

That is the main reason why podman exist, so then one can accomplish that task without an extra daemon, dind, nor using the host's dockerd daemon.

This way you don't need to grant "docker" group to the "ci" user and you avoid having your cluster compromised one commit away :)

You need more constraints than that to explain why you can't just mount /run/docker.sock in the CI container.

If you mount docker.sock you're building on the runner's daemon so you have to worry about builds interfering with each other. If you use docker-in-docker you get a clean environment every time.

If you don't manage your own CI server then that is not an option is it?

I also never said that it requires two levels of Docker or that it requires DinD. I was responding to the general question of the parent who was asking a question of someone who was running DinD. I responded to the GP below that Kaniko also solves this problem so clearly I'm not advocating for running DinD or that this is even needed.

As mentioned elsewhere in the thread, this does not require DinD. You can connect to the host’s Docker daemon. (Other interfaces such as K8S are also much safer.)

Isn't exposing the host Docker daemon the security issue that people grouse about?

Yes it is. The Docker socket - /var/run/docker.sock is owned by root and so anyone that has access to it effectively has root on that host. User namespace mitigate this to some degree.

Yes and I am one of those people who mentioned elsewhere that this does not require DinD. Please reread the question I am actually responding to which is a general questions "What kinds of tasks ...?" The parent is asking what 'problem is the DinD solution is solving for. You also can not connect to the host's Docker socket if you do not manage that host where Docker is running in the first place.

“kind” is a example in my day to day, how do I run an ephemeral kube cluster in ci to deploy and test my app against without dind/kind? I want N clusters running on 1 host to support many CI jobs.

https://github.com/kubernetes-sigs/kind https://github.com/bsycorp/kind

FWIW, Kind actually uses CRI-O (same underlying backend as Podman) for the inner containers.

Ansible Molecule tests depend strictly on Docker: https://molecule.readthedocs.io/en/latest/

Also if your application is shipped as orchestrated containers (like docker-compose), or as multiple containers in a 'pod' (e.g. sidecars), you may want the ability to run containers from containers as part of CI.

As of Molecule 3.0, at least -- they didn't in the past.

A while ago, I, unfortunately, decided to add "proper tests" to all of my Ansible roles and decided to use Molecule (2.22, at the time, IIRC). As I don't use Docker, I was using "lightweight VMs" I had created (w/ Packer, converted into Vagrant boxes) with VirtualBox for all of this testing.

I spent I don't know how many hours across several days learning the "toolchain", getting everything setup and working properly, adding full test coverage, and so on. Not long afterwards, they released Molecule 3.0 which required using Docker. :/

We use docker-in-docker as well to deploy our Jenkins workers. It has it's pros (running Ubuntu 18.04 and Ubuntu 20.04 workers side-by-side and ease of pushing out an update) but it's fragile and confusing. Things can get really hacky when you need to start sharing volumes or injecting additional configuration.

I'm much more confident with packer now though. Next time I do any major work on our Jenkins infrastructure I'm ripping out docker-in-docker for the workers and replacing it with packer built images.

We docker-in-docker too, exactly as OP describes. In jenkins CI/CD.

> but is intended for the developers of Docker to debug Docker itself

Heh. We use DIND with docker compose to have a container which has KUBERNETES inside. And it even works.

How's that for a wild use-case?

EDIT: That's done to create a local dev environment, with K8s, localstack, infra, etc. Instead of having multiple machines or deploying everything outside containers.

I also needed to run docker commands from within a container (Jenkins container executing docker cmds on the host it's running on). I don't know how common this is but I had the same use case.

I do a lot of build automation work, and running containers within containers would be super useful for me.

That said docker-in-docker doesn't work without running privileged or forwarding the host port.

It's a non-starter for me, there are the obvious security problems, but also practical non-security issues. Forwarding the port causes encapsulation issues, a build job can finish leaving stuff running, can also interfere with other jobs on the system.

Using privileged containers isn't an option when using things like ECS fargate.

There are some reasons to use docker in docker beyond debug docker itself. For example, in Jenkins case, select specific proxy settings for the daemon for a specific job without affecting other jobs in the same node.

Heavy compose user here, and when I tried the podman equivalent of compose was missing some functionality that made me move back.

I had the same experience, and I'm not really sure why I can't use docker-compose with podman. If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal?

I did try that and it doesn't work, and podman compose isn't as useable. So I switched back to docker.

>If it's a drop in replacement, why can't I alias podman as docker and use docker-compose like normal?

Podman is (mostly) a drop-in replacement for docker. However, docker-compose is a separate package from docker that requires explicit installation. So too is podman-compose separate from podman, though unfortunately it still needs more work.

I can explain this. Because docker-compose talks to the docker socket and uses the docker protocol. This was the last piece that we nailed with podman. As of Podman 2.0 with the new REST based API with a Docker compatibility layer, we are very close to just being able to use the docker-compose binary to talk to a socket (ran by systemd) which fires up podman.

It doesn't quite work yet today because we are still implementing some of the REST verbs, but it's close. It's definitely a strategic direction for the roadmap. Stay tuned.

I've found your issues with podman to be a theme for Red Hat projects in general. Have you ever tried to configure Keycloak or OpenShift, for example?

Spend too much with Keycloak lately - it's more like a framework with some basic batteries included and you'll hit into corner cases pretty fast. If you are fluent with Java, their templating engine and all the other funky stuff it's a powerful tool - but i.e. their REST-API is bad. List all users including groups? That'll be an additional HTTP call for every user.

Their tools solve complex problems and are free to use - so I'm thankful that they exist - but I can't help but think that there is some lack of elegance&design that causes lot's of complexity - there is no "unix philosophy" to abuse different tools or components to solve problems - it's more like either you'll use the high level APIs with certain non-obvious assumptions (i.e. worked for us, good luck :) or feel free to hack on it if you grok our complex low-level frameworks and libraries... went after a NetworkManager bug once and it was a tour de force between c, glib, dbus with zero documentation. systemd and Keycloak feel very similiar. Powerful if you fit their usecase - horrible if need to tinker with it. But to be honest I've got no idea how to solve these complex problems otherwise. It's probably the best we can do at the moment. Or are there any non-cloud/non-sass solutions that actually have all the features?

This might be biased on my part but experiences like this is why I tend to openly avoid most Red Hat projects in general. There are definitely some great projects like RHEL, Ansible that are nice to work with.

Even professionally, I tend to steer clear of immature Red Hat projects.

We evaluated Keycloak but went with a vendor solution. OpenShift I believe was also evaluated at my firm and hit a dead-end.

Openshift is a great solution for a k8s setup on prem. From my experience it doesn't bring a whole lot when you're already on the cloud. All the additional services openshift provides, are already available as part of the major cloud offerings.

Openshift years ago at least (last time I looked) - SHUDDER! I couldn't believe it was what big business / enterprise was standardizing on as best in class. It looks like it's now Kube / Docker connected, but back then I think it did it's own thing?

The cost in time / $ / config to maintain and operate at the time was not pretty (all this no doubt has changed a lot).

They gave you three pods for free until they moved to Openshift 3. I ran an entire startup five years using them without paying a single penny. A nice Heroku alternative. (Iot control server, so basically no load)

Openshift was very generous to startups, I can back that up as well.

It always seemed like added complexity to me in exchange for free hosting. I liked the idea in concept more than in practice. But we also weren't big enough to really justify it so eventually we switched to simpler VPS hosting.

Same, I ran some stuff for free until the switched the mandatory pay plan, then moved to cheap vps

Just spent the last two days messing up with Keycloak on kubernetes... I feel you.

Your life will be a lot less painful if you abandon docker-in-docker. The Jenkins solution is to simply execute containers using the host's Docker daemon, or via a container orchestration system which you call out to.

The latter is the best option as it's much more scalable and doesn't require ugly proprietary hacks. Docker in Docker requires vertical scaling and more complex management for intermediate states/maintenance.

Podman probably isn't ready to replace Docker yet, but rootless containers are the easy-mode for federation of clusters. If you can supply the other features you need (and I think most can) it's probably worth it in the long run.

I love the idea of Podman being daemonless and (potentially) rootless. Regardless of the presence of security flaws in dockerd, the Podman architecture just feels cleaner.

And the CLI compatibility is great. Until it isn't. At work we switched to Podman for a small deployment because Docker didn't yet work with cgroupsv2 and many hours were spent debugging Podman-specific issues. In the end switching to cgroupsv1 would have been significantly less work.

Therefore claiming you can `alias docker=podman` is a bit disingenuous. You can, but only if you don't do anything Podman can't handle and what it can and cannot handle isn't immediately obvious.

All that said, I wish the project the best and hope it reaches the maturity where this alias actually does work.

Genuinely curious what issues you came across. Any chance you documented them somewhere?

I'm afraid I didn't and I cannot really recall, but many were related to networking. I think one was that I couldn't have two containers on the same network listen to the same port, another one was related to Podman not playing nice with nftables, but the specifics elude me.

EDIT: Upon further reflection, I think Docker doesn't really work with nftables either, so that one isn't on Podman. It just so happened we made that switch at the same time. Regardless, there were other problems. I'll check to see if I can find any records of the problems later.

I was just about to complain about another thing to learn. Then I saw you can do "$ alias docker=podman". Just want to acknowledge the importance of that work. Making things compatible is both boring and a pain, but it's a door opener for people like me who refuse to learn your new API because I know a decent one already.

Completely agree.

One thing I want to point out though for anybody not familiar with the differences between podman and docker, for the most part alias docker=podman will "just work" except for these situations:

1. docker-compose. podman-compose attempts to cover this but I've heard it's not quite there yet

2. Mounting the docker socket into the container. Podman is daemonless which means that won't work. There is work going on right now to allow Podman to be driven in a similar way if needed, but I recently tried to set it up and hit a bug[1]. CRI-O brings a daemon, and it is used extensively in Kubernetes and OpenShift, but not so much outside of that.

[1] https://github.com/containers/podman/issues/8323

It’s now possible to mount a docker compatible socket inside the container using podman.

Can you provide more information on this? A blog post or some instructions? I'd like to read more.

So far, it's just a less functional docker and didn't bring much to the table. I don't feel the need for daemonless as docker has been working fine.

I've swapped back to docker on Fedora and CentOS by forcibly installing docker on those unsupported platforms because podman-compose doesn't work, containers just didn't work like expected in several situations (forgot detail) and portainer wouldn't work against it.

Either they should have sped up the podman development or shouldn't have deprecated docker at this point. Maybe RedHat just didn't like a third party taking the stewardship in the container business.

Plus there's no Buildkit support.

Unless you're optimizing for stagnation, this really is a terrible approach to reaping the benefits of innovation. If your prereq to trying something better is that it needs to mirror something that's "decent already," you'll spend most of your life stuck on the same plateau.

History is full of technologies that are superior to existing options on one metric or another and yet failed to take over due to switching costs. Attacking someone as “optimizing for stagnation” is a really bizarre way to view their reaction to a product offering a smooth on boarding process.

Yeah, you're right. I should make the time to learn more stuff and never make excuses for why I don't.

On the other hand, I don't think you fully understand where I'm coming from. "Optimizing for stagnation", "stuck on the same plateau". Idk, I just want to run a container. I'm totally fine being on the same Docker plateau if something like Podman makes me re-learn what I already know how to do.

Unlike Git, which, from the start, was different and BETTER than SVN, can you really make the case that Podman would be BETTER if it didn't try to follow Docker conventions? If not, what are we talking about here?

We've tried this (podman 2.0.5) and have hit some really really annoying bugs:

- Build layer cache doesn't seem to work. If I rebuild locally with podman, it correctly detects cache hits and the build is fast. On our Jenkins server (RHEL 8) with podman 2.0.5 it doesn't. It randomly doesn't cache hit, causing builds to take 20x longer than with docker CE.

- Podman is insanely slow at building images in general. COPY emptyfolder/ /emptyfolder/ takes 2 seconds. We have dozens of things to COPY and it's stupid slow compared to docker CE. Buildah doesn't seem any better.

- Systemd integration has bugs. If you use the default generated systemd unit file, it does not kill processes when exiting and leaves them dangling. Even after removing the strange KillMode=none it says to put in there, it still leaves processes dangling. Podman sometimes loses track of the container. It will list nothing in "podman ps" but the processes will still be running.

Are you trying to use cache-from? This is explicitly not supported:

> --cache-from

> Images to utilize as potential cache sources. Podman does not currently support caching so this is a NOOP.

Not using --cache-from, though I did try the various caching command line options to try to troubleshoot. Just a vanilla Dockerfile with a few RUN commands and COPY commands. It would hit cache until about the third layer, and then would cache break every time.

On my local computer (Arch) podman is v2.1.1, which seems to have whatever bug I was hitting fixed.

So I guess my complaint isn't about podman specifically-- It had bugs and they were fixed, and that's great. But I hate that RHEL 8 touts it as a docker replacement, and won't carry docker in their repositories, when the version they have in their production releases is so broken.

We eventually sledgehammered docker CE's CentOS repo into our RHEL 8 jenkins server and now everything works perfectly.

Running podman on the production webservers seems to work okay though-- apart from the process killing problems.

I've said this so many times, but we wanted to carry Docker in RHEL 8, but we couldn't. We would have been forced to use Moby and it's not a well contributed to project anymore. The whole Docker contributor community has fallen apart, and Red Hat needs to be able to support a container engine until the mid 2030s.

You can get access to Moby in Fedora, but it just wasn't viable to include docker in RHEL 8 for both legal and community risks.

copying things might be slower if you have a .dockerignore file. try without the .dockerignore file.

I find the podman integrates with systemd well claims a bit dubious.

Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background.

I found this very surprising. I still don't understand why they made that choice.

At least they do play nice with the whole "systemd owns the Cgroups tree" story. A thing that was always a bit painful with docker.

Fun note. Systemd-nspawn actually can run OCI containers directly as well these days. However I'm not sure if it's feature-complete

In an ideal world, systemd-nspawn would be the preferred CRI. That was basically the goal of rkt.

What happened?

Perhaps that choice comes from wanting their target platform to be linux, rather than systemd?

Hopefully most of you who have had issues with Podman have opened bug reports or issues. We are working to fix incompatibility bugs and getting out new releases all the time.

As far as docker-compose support, in Podman 2.0 we have added APIV2. This is a socket activation REST API. This API has a compatibility mode that implements the "Docker API" meaning podman can be setup to listen on the docker.sock and launch containers. We also have the more advanced Podman API for support of concepts like Pods. The API should be able to work with docker-py based scripts and Compose. We are getting lots of community support in fixing up our inconsistencies.

This is a fully open source project and we love to get contributions.

I always like to chime in and recommend people check out Singularity[0] containers if they haven't before. It's more common in academia because you can run containers without privileges which is nice in HPC environments. The containers themselves are simple immutable flat files that can be easily copied around, backed up, etc. There's also no system daemon. It's nice knowing everything you need for your container system exists in the containers themselves and a single singularity executable. No volumes, remembering to append --rm to avoid dangling containers, wondering where the actual images are stored, etc. Of course there are tradeoffs but I like them for certain use cases.

[0]: https://sylabs.io/

If you are running podman rootless it will use the fuse-overlayfs driver which comes with various limitations and is much slower than Docker:

* you are limited to 1024 FDs for your entire container. So running make -j40 usually ends up with various errors due to running out of file descriptors. You can of course raise the limit for your own user, but this may not be trivial on a shared system.

* getting podman to obey a raised user FD limit is non-trivial on CentOS 8. podman is not well supported on CentOS, you need latest Fedora if you want the various bugs/limitations fixed.

* fuse-overlayfs is a single process for the entire container and quickly becomes the bottleneck for any IO intensive operation (e.g. don't try running AFL inside podman, it'll be way too slow and peg the CPU at 99% running fuse-overlayfs).

Fuse shouldn't be necessary though if I could just give it access to the files that my user has access to (of course you'd loose the snapshotting/committing ability), perhaps using -v would improve performance?

Using docker on CentOS 7 was a lot easier (and faster) than using podman on CentOS 8.

For anyone not using Podman just because lack of docker compose like functionality, there are multiple alternatives.

* Podman's official stance is to use Kubernetes YAML. The provide tools to generate this as well as convert docker-compose to kubeyaml

* podman-compose is not official, but it works well. you have to note that its not 100% compatible with dc but a lot of dc files run out of the box without modifications

* ansible with podman

* i personally use the podman cli with makefiles to get docker-compose like functionality

* also, podman v2 supports a docker comptaible rest api so its possible that docker-compose can be modified to support podman in the future

podman-compose works really well IMO.

For my stacks, the only thing i can't get working is routing a specific container through a host defined network (not a container). I'm certain podman can do it, i just haven't had the time to "fiddle" with it.

With docker (compose) i just define an ipam network (external) matching the host adapters network, and connect the containers to that network, and the spice begins to flow.

I pray for a Hashicorp Nomad driver as well!

Hi, it exists <https://www.nomadproject.io/docs/drivers/podman>! This has gone right passed me.

I find the podman integrates with systemd well claims a bit dubious.

Last time I checked both podman and CRI-O double fork and have reimplemented process supervision from scratch (through conmon) whilst they could get all those features for free if they didn't daemonize themselves and let systemd handle running things in the background and do the process supervision.

I found this very surprising. I still don't understand why they made that choice.

Looks like you may have double posted. Not complaining, just letting you know in case it was unintentional: https://news.ycombinator.com/item?id=25166141

Podman is amazing for CI and dev environments:

* It doesn't require special privileges to run

* It runs containers as the same UID as the calling user, so directories mounted into the container can't be polluted with files belonging to other UIDs, as Docker tends to do

* It's easy to get it to only pull images from custom registries. With docker, this requires some fiddling (at least last I tried).

The only thing missing is Debian packages in Debian Stable (currently available in Bullseye aka testing).

I like the daemonless architecture a lot, but until there's a quick and painless way to install it on OSX and Windows developer machines, you're going to see very limited uptake.

Maybe it doesn't count but podman works great on WSL2 as far as I can tell.

I actually like the idea of Docker as a better systemd (or rather, the interface is better). No bespoke file format, programmable API, no need to google for the right journalctl switches, and then of course the advantages of containers and images over processes and system packages. I’m not suggesting everything should be a container nor that docker is the ideal implementation, but it certainly points in the right direction.

> I actually like the idea of Docker as a better systemd (or rather, the interface is better). No bespoke file format, programmable API, no need to google for the right journalctl switches, and then of course the advantages of containers and images over processes and system packages. I’m not suggesting everything should be a container nor that docker is the ideal implementation, but it certainly points in the right direction.

Docker was created to give an extra life for legacy applications depending on outdated packages, the concept of gluing all dependencies withing a compressed rootfs and shipping that, with low effort.

Docker became popular because of that, but seems that did more harm than good for the ecosystem in terms of security, that why podman might have a great future. The docker interface is bad and comparing that with systemd is quite a stretch. ;)

Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.

Aren't containers just processes with namespaces and cgroups?

How does podman improve on container ecosystem security?

Check the first link in the "resources" section at the bottom of the article page.

Right. I misread your comment to imply that podman improved on the "legacy applications depending on outdated packages" problem.

I suppose a more secure runtime doesn't hurt.

> The docker interface is bad and comparing that with systemd is quite a stretch. ;)

I don't know, systemctl is always a pain to interact with. The Docker CLI is pretty intuitive--you have different resources and different verbs for interacting with those resources. This includes logs. No need to use a separate (and confusing) tool nor dig through man pages. I'm of the philosophy that it's better for a tool to be intuitive than obscure-but-has-manpages.

> Anyhow, aren't systemd units ini files? The journalctl ships with man pages though.

I guess I meant that INI isn't a standard file format--different parsers behave differently, and structured data is often coded in strings in some bespoke format.

> Aren't containers just processes with namespaces and cgroups?

My mind isn't made up that containers are the ideal process unit, but I do like some things about them (and the container ecosystem more generally). That they come with their dependencies bundled is pretty nice, but I think the toolchain needs to improve to mitigate security concerns and so on. Something like Nix would be helpful, but Nix also has its own problems. Again, it's directionally correct. In any case, users don't need to be managing their own namespaces and cgroups.

But systemd has all of those things. Everything in systemd can be orchestrated with dbus. In fact that’s all systemctl really is — a dbus client. And the file format is ini. And shell style environment files if you want.

And systemd actually comes with two image launching systems systemd-nspawn and systemd-portable. And then with systemd-machined you can add software that needs virtualization too.

The interface to journald is more complicated than it should be but it’s also really powerful — docker logs doesn’t hold a candle to the kind of filtering it can do.

Overall systemd is a stupidly powerful and featureful supervisor compared to Docker. Just the dependency management alone should demonstrate that. Then you get mounts, swap, socket activation, more powerful restarting policy and the whole suite of isolation and security features.

> The interface to journald is more complicated than it should be but it’s also really powerful

I thought the point of logging to stdout (i.e. docker logs) is that you just take that output and dump it to another server for processing and filtering.

journald/journalctl seem to be a solution a few decades late to the party. For a single user machine or a single app prod environment, I would take a plaintext log any day of the week. At least I can remember how to grep the damn thing. And then when you get to a distributed system, what's the point of journalctl? You would hopefully have all of that logging aggregated together in one place with a much nicer web interface for it all.

Exactly this. I just want a stream of logs that I can process with ordinary unix tools without jumping through hoops. I don’t need any extra power for its own sake.

If that’s all you want there’s really no hoops.

   journalctl | grep
Does what you would expect, it will happily (and by default on most distros) forward to syslog if you want files.

The big ease-of-use win for journald is that it captures process stdout. No need to run daemons in the foreground ever.

close, but it’s actually

    sudo journalctl -u $service -f | grep
Not a huge deal, but I have to google it every few months because I don’t use it very often.

Note also that if I just want the log stream, I have to pipe through less to get the full log messages. There’s also a flag for it, but I can’t remember my workaround is easier than digging through man pages.

Again, no big deal, just friction. Like everything in the systemd ecosystem—everything is manageable, but it’s tedious.

How are you using Docker that doesn't need a bespoke file format? You're building from a Dockerfile and either scripting around the docker command or running docker-compose.yml


I would argue that docker-compose.yml is every bit as bespoke as a systemd service file (so, "medium"; a custom schema embedded in yaml/INI).

Like I said, I don’t actually want docker or docker compose, but they are directionally correct. Notably YAML over INI is a nice improvement. Also, that the docker daemon has a real API and not just dbus.

Well, Docker has an API and its own file format (the Dockerfile), I constantly have to look up Docker switches as it's not a tool I use every day and it's not at all intuitive, and I'm still unclear why people love this layered/stack image approach given how unbelievably slow it makes everything. Apps that would once have been a 50mb tarball are now routinely a gigabyte of downloads, perhaps because whatever random version of Ubuntu I downloaded previously is now slightly different? Who knows.

There is, frankly, a lot of appeal to me in a simple INI file that starts up something I unzipped.

Last time i did look into docker it couldn't even tell if a service is up or still starting and there were all kinds of weird hacks around that like "is something listening on the configured port". Did that improve?

Systemd struggles with this as well. While a service can easily tell you if it's "up", sometimes the only way to determine if it's "available" is to check. Keep in mind that the sentinel for any given service will be different. If you think about it, how could any external monitoring service determine when a service under it's purview is available without actually checking?

Maybe it already has something like this, but I feel like systemd could provide an API endpoint for applications to send a simple status when they are "ready" - at which point it would be up to the developers to provide that at the right time.

From the link: "systemd-notify may be called by daemon scripts to notify the init system about status changes. It can be used to send arbitrary information, encoded in an environment-block-like list of strings. Most importantly, it can be used for start-up completion notification."

Perfect. Thank you!

You're welcome :)

Am I understanding this wrong or is HEALTHCHECK supposed to enable that?


It now becomes clear that podman is a new way of folding docker into systemd.

at some point systemd is an embrance-and-extend formula to control the linux ecosystem

I was just thinking the same thing, and was going to ask if there was anything like Docker Desktop, something to make it simple to use on Windows/MacOS!

Before there was Docker Desktop on Mac there was Docker Machine.

And of course there's a Podman Machine: https://developers.redhat.com/blog/2020/02/12/podman-for-mac...

I find it somewhat surprising that we haven't seen more adoption of the redhat docker (lowercase d) replacements after Docker (uppercase d) started pushing some of their monetization efforts.

The conclusion that I have to reach is that there are more docker users on Macbooks than I realized.

> One of Podman’s greatest advantages is its complete CLI compatibility with Docker. In fact, when building Podman, Docker users can adapt without any significant changes. For example, you can use the alias command to create a docker alias for Podman: `alias docker=podman`

Wow that's a great adoption hack; just alias another program as another

Would that be covered by the same issue as Google v Oracle?

Maybe, but it wouldn't matter if they're OSS to begin with

That's true, don't know why I didn't think of that to begin with, just saw them as "products" from two companies.

Last time I tried podman it looked very cool being rootless and daemonless, except that since it's rootless I couldn't create the necessary network interfaces... making it pretty much worthless.

Podman 2.1 supports CNI networks (`podman network create`), so it might be worthful now

Great! I'll check it out!

So run it as root then? Am I missing something?

If GP thinks the selling point of podman is that it doesn't have to run as root, but to actually do anything meaningful they have to run it as root after all, why would they use podman?

A rootless docker would greatly simplify some use cases... For example, no need to do docker in docker.

Edit: apparently now net4slirp is recommended... Maybe I'll give it another try.

The other major benefit of podman is that it doesn't use a daemon.

Many people say that, but don't feel like it's very important.

Having a deamon or not is a technical detail that most people do not care about in my opinion. And it has advantages too, like accessing Docker remotely or from another VM on the same host, or directly from the host which is nice for Docker on Mac or Windows.

See replies here [1] for more information

[1]: https://news.ycombinator.com/item?id=25165789

These issues are because the deamon is root, not because it's a deamon.

After reading this thread, I realize that the 4 GB (average) of my vagrant box images wasn't that bad in retrospect, and that loaded a fully baked OS with packages from apt and just required VirtualBox and its extension pack. (this was not an optimized setup, I was such a noob at the time, just trying to wrangle our developer tools)

The only reason I moved to docker years ago was I wanted a tighter reproducible workflow (ie docker-compose up) for all other developers, turns out vagrant has (at some point in the last 5 years) solved this problem too, with packer (I didn't know about this 5 years ago for whatever reason)

Makes me long for going back to that. At least once you built your base image, it was done and you could just make fast linked images from there. I bet with Alpine you could get a vagrant up in a few minutes tops, then it boots in seconds.

Never liked Ruby as a config language, but doing complex things like setting up shared networks and folders was a breeze comparatively I felt.

Never used vagrant in a production capacity though. Everywhere I ever worked always deployed to bare metal or essentially we bought our services (cloud functions, semi-managed containers etc)

Ditto. I for one would love to have Vagrantfiles-like Ruby scripts in Docker. Or even Typescript.

So many setups would be achievable with simple ifs.

Daemonless and rootless are killer features.

The lack of filesystem isolation and volume support are the last things keeping me from jumping ship.

Rootless is coming to Docker as well since containerd switched to cgroups v2, which was a requirement for rootless

Docker has been supporting rootless mode since 19.03

Podman already supports `podman volume create`, though it doesn't support volume plugins.

Meta: there seems to be two kind of comments here...

1) why the hell would you want Docker-in-Docker

2) I can’t live without Docker-in-Docker

1) people who haven't used docker very much 2) people who are using docker to do literally everything

I think there's a disagreement about what Docker-in-Docker actually is. Some people think it's running a docker daemon inside a container, and others think it's accessing the host docker daemon from inside a container by mounting the docker socket as a volume

Sometimes I feel grateful having let all these ephemeral things things pass by without having needed to learn something that was going to stop being "in" so soon.

Containers are going to be there for a while and while the engine running the containers is not always the same, the command line Docker client and the Dockerfile DSL seem to become the standard for Linux containers.

I've used Podman, SystemD and Buildah.

It's great to hack around with. SystemD is particular is very polarizing and I don't want to start a flamewar, but it helped me "get" init systems.

Podman seems to have a better security model than Docker, so we were trying it out at work too.

Redhat is clearly very invested in this.

There was excellent documentation and good tooling around all this.

Does Podman let you get the real SHA-256 digest of the image, pull it to a machine, and then launch the image only if it matches the specified SHA-256 digest?

The podman 'create' command docs [1] do not list a '--digest' argument. I found no example of specifying the digest as part of the image name.

Docker does not support this. You can get an image file and calculate the SHA-256 digest of it. But Docker does not let you say "start a container using this particular SHA-256". That's because the SHA-256 that docker uses internally is just a hash of dockerd's internal metadata about the image. That metadata is different on every machine. I felt extremely disappointed when I discovered this. And I lost a lot of respect for the Docker developers.

If you want to deploy based on image hash with Docker, you must add your own verification step before creating each container instance. Terraform's docker plugin does not perform this step. And the local dockerd will not check the SHA-256 on reboot. If you just follow Docker's "best practices" and use the provided tools, all of these sources will needlessly have root on everything you deploy:

- Your docker image repository

- Any machines with credentials to push to your docker image repository (CI system, engineer machines, anyone with access to engineer machine backups)

- Anyone with permission to push any public image that you use. The way most CI systems are configured, all jobs get the same credentials. So this includes your build and test jobs. That fancy linter Docker image you're using? Anyone who can push to its Dockerhub account can replace the image version you selected with one that roots all of your systems.

- Dockerhub itself

Why would RedHat switch to a Docker replacement without this crucial security feature?

Am I missing some important point that makes all of this ok?

[1] http://docs.podman.io/en/latest/markdown/podman-create.1.htm...

Podman, buildah, and skopeo are insanely awesome tools that let you transition an image from a public registry to your own private deployment in a jiffy. The article doesn't mention it but podman can also dump a pod configuration you can use in kubernetes and open shift/okd so if you're not a yaml wizard you can still get stuff running containerized.

Nope, it's still not there, simplest things work but as soon as you diverge to something more complex it fails.

I tried using it as Docker replacement, but various tools that use docker (using dockerized pip in serverless framework) and complex docker-compose files (dockerized Magento) were broken.

I really struggled with podman, a bunch of containers (I cant remember which ones) didn't work, and build tools choked as well.

At work we dont get root access so I thought it would be perfect but sysadmins couldn't figure out how to configure either so was bust there too.

I've also personally struggled a lot with this at $WORK, but some of the new features in RHEL 7.8+ (specifically GA'ing of the fuse-overlayfs graph driver) have actually alleviated a ton of the issues I've seen previously. If you haven't tried that I'd certainly recommend it, YMMV

I just installed my new $WORK docker-compose using podman and podman-compose, and it worked out of the box. To be fair, I failed at it one time before at previous job... So... Coin flip?

I had a repeat of the same experience too recently. It was very satisfying having it work now.

I had the opposite experiences, everything worked out of the box.

Any docker Podman corner cases that people have run into? I like the idea of rootless containers but I can do that ether easily by adding one line to my dockerfile or adding a user flag when I run a docker container. What other advantages am I getting?

This isn't about root inside the container but rather it doesn't need root on the host system.

Not sure if you count it as a corner case, but podman lacks a docker-compose equivalent - last I checked, a few months ago, there was only a community port project that still needed a lot of work.

Not really corner cases, but compatibility from rootless podman to docker ends where privileges are needed, e.g., in the case of networking.

No need to run a daemon. Security is better.

If you lock down the docker user, is it a problem?

The biggest issue lies not in the few lines in /etc/passwd or /etc/group, but rather the highly privileged process with a large attack surface that is the Docker daemon.

Generally, it's been pretty easy to transition for me, but I used to rely on things like the automatic nginx image that you need to mount the docker socket into so it can route traffic to containers when they come up. Since there's no daemon, there wasn't a way to do that. I think people are working on it, but I haven't tried recently.

Work also uses a special docker RUN command that can use the host's ssh keyring to, eg. Install something from a private github repo. Doesn't seem to exist in podman.

I use that nginx image too, and it's the one thing that keeps me using docker for some small single-node setups. For basically everything else, Podman is a better experience. Now that I've discovered Podman can run K8s Pods (and now Deployments) I use it all the time. I don't know how I lived without it!

> Podman can run K8s Pods (and now Deployments)

That is amazing; thanks for pointing it out:)

I'm an enduser and I expect things to "just work". That seems to match a lot with Red Hat's Linux distros (i.e. RHEL): "Just use it, it's a defacto standard so most third party software is tested on it, don't worry about underlying components as Red Hat has made sane choices for you."

That's the theory (and Red Hat's marketing). I wish I could just be using CentOS/RHEL and not worry about the choices Red Hat makes. But the shit is piling up (sorry for my harsh words, no offence intended).

So, let's containerise something simple on top of the "centos" container image, let's say postfix. Oh, postfix on centos needs systemd for logging. But systemd in a container is a nightmare. You can get systemd to work in a container if the host system is using containerd and you pass certain special files from the host to the container. So just use Podman instead of Docker, right? Podman has functionality to make systemd in a container work. OK, I can run my container on my fat workstation (a Centos system with Podman). But is it still portable? Can I just run it in Github Actions, AWS ECS, a Windows machine with Docker Desktop? Nope. It's not portable.

Let's put the systemd rant aside (I really like the systemd CLI UX, but the architecture (dbus, etc) seems to only serve one usecase properly (Desktop systems) and seems to be heavily wrong for the container usecase).

Let's rant about the architectural choices Red Hat is making for containers. Before Red Hat started to get involved there was the open source, (now) cross-platform containerd that a lot of tools are building on top. It consumes the low level runtime runc and both provides an API for Kubernetes (CRI) and additional things. It is highly pluggable and therefore the "only container runtime you'll ever need". And it's purely community governed (CNCF).

What is Red Hat doing? Are they building their container tools on top of containerd? Nope. They create their own low level container runtime (crun), their own mid level container runtime (cri-o). But cri-o only covers what's needed for Kubernetes. So to be able to build container images, etc (the "working with containers on a single machine" usecase) additional tools are needed (podman, buildah, skopeo) and they have to implement the missing functionality themselves. So Red Hat's Open Shift (Kubernetes distribution) builds on an entirely different stack than Podman.

Fragmentation everywhere. Was that really necessary?

Why should I care as an enduser? Well, I can inspect and debug all tools that use containerd the same way (i.e. using the "ctr" tool). I can relatively easily even write my own tools that use containerd (via it's grpc api) and can access the resources (containers, images, etc) that these tools are managing to solve my special custom needs.

For the Red Hat stuff everything is different. I could work with cri-o using crictl and write custom tools using cri interface. For Podman I would need to use Podman's API. (Which has annoyingly entirely changed between v1 and v2).

ok, enough rant (I could go on with how hard it is to get an up to date version of podman on a supported RHEL system even with app streams, that UBI images don't provide podman/buildah/skopeo, etc etc etc).

Please don't mistake me: Podman, cri-o, crun, open shift are all amazing technologies. And it's great that they are mostly open source via upstream projects. But I wish this whole fragmentation hadn't happened. For me as an enduser it's nightmare.

and yes, people have proven that it's possible to build something like podman or buildah on top of containerd / buildkit! No need to reinvent the wheel!

https://github.com/rancher/k3c https://github.com/genuinetools/img

Assuming that it's Red Hat's fault that there is no el8 package for Docker, I think Red Hat is really shooting themselves in the foot by not supporting Docker in RHEL8. (You can still install the el7 version of Docker on el8 by performing an enable/disable/enable/disable/... module dance, but... why isn't there just an el8 version?)

Red Hat ships podman for RHEL8, that's what the article is about.

Yes, but not Docker. (Also Podman is developed by Red Hat.) I think they have rather high hopes there, and are more likely to just shoot themselves in the foot by being one of a small number of distros that don't have Docker. In fact, rebuilders of RHEL (e.g. Oracle Linux) should maybe consider explicitly supporting Docker. "We are 100% compatible with RHEL but additionally also support Docker" definitely makes for better marketing copy than just "We are 100% compatible with RHEL".

I'm not familiar with Podman but I welcome any alternative to Docker.

And so it starts, the journey to something else. Thanks Docker.

Does it support GPU when installed on Windows 10 WSL1?

Podman is really a nice piece of software, we are using it successfully on debinan 10.

I am still thinking whether to include Docker or Podman in my book[0]? Since I am leaning towards RH tech (like SELinux), I think I will go with Podman. I think it's the future on Fedora-based systems.

[0] https://deploymentfromscratch.com/

I'm finding podman great for my (not actually that complicated) docker use-cases

Where can I find documentation about the security benefits claimed for podman?

Look for blog posts by Dan Walsh.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact