
Kaniko: Build container images in Kubernetes - dlor
https://cloudplatform.googleblog.com/2018/04/introducing-kaniko-Build-container-images-in-Kubernetes-and-Google-Container-Builder-even-without-root-access.html
======
jacques_chester
This work (and related efforts like Img and Buildah) is a big deal.

Right now docker images and Dockerfiles are joined at the hip to the Docker
daemon.

It works great for local development, but for hosted systems that run on
containers, it's a dire mess. I have personally slammed head-first into
Docker-in-Docker quagmires on Kubernetes and Concourse. Not knowing the
particular arcane rites and having neither sufficient eye of newt nor
sufficient patience to get it to work, I like everyone else in the universe
gave up.

Not an acceptable state of affairs, given the many problems of Dockerfiles in
themselves. Dockerfiles force an ugly choice. You can have ease of development
or you can have fast, safe production images. But you can't really have both.

Kaniko is another step in the direction of divorcing docker images as a means
of distributing bits from Dockerfiles as a means of describing docker images
from Docker daemons as a means for assembling the images. All three are
different and should no longer be conflated.

Disclosure: I work for Pivotal, we have a lot of stuff that does stuff with
containers.

~~~
derefr
> Not knowing the particular arcane rites and having neither sufficient eye of
> newt nor sufficient patience to get it to work, I like everyone else in the
> universe gave up.

One thing I feel like more people need to know: Docker container-images are
really not that hard to build "manually", without _using_ Docker. Just because
Docker itself builds images by repeatedly invoking `docker run` and then
snapshotting the new layers, people think that's what their build tools need
to do as well. No! You just need to have the files you want, and know the
config you want, and the ability to build a tar file.

Here's a look inside an average one-layer Docker image:

    
    
        $ mkdir busybox_image; cd busybox_image
        $ docker pull busybox:latest
        $ docker save busybox:latest | tar x
        $ tree
        .
        ├── 8ac48589692a53a9b8c2d1ceaa6b402665aa7fe667ba51ccc03002300856d8c7.json
        ├── f4752d3dbb207ca444ab74169ca5e21c5a47085c4aba49e367315bd4ca3a91ba
        │   ├── VERSION
        │   ├── json
        │   └── layer.tar
        ├── manifest.json
        └── repositories
    
        1 directory, 6 files
    
    

• `repositories` contains the tag refs that will be imported when you `docker
load` this archive;

• `manifest.json` contains the declarations needed for the daemon to unpack
the layers into its storage backend (just a listing of the layer.tar files,
basically);

• the SHA-named config file specifies how to reconstruct a container from this
archive, if you dumped it from a container (and I believe it's optional when
constructing a "fresh" archive for `docker load`ing);

Each SHA-named layer directory contains:

• a `layer.tar` file, which is what you'd expect, e.g.:

    
    
        -rwxr-xr-x  0 0      0     1037528 16 May  2017 bin/bash
    

• a `json` file, specifying (the _patch_ of!) the container config that that
layer creates. (If you're composing a docker image from scratch, you just need
the one layer, so you don't have to worry about the patching semantics.)

That's pretty much it. Make a directory that looks like that, tar it up, and
`docker load` will accept it and turn it into something you can `docker push`
to a registry. No need to have the privileges required to run docker
containers (i.e. unshare(3)) in your environment. (And `docker load` and
`docker push` work fine without a working Docker execution backend, IIRC.)

~~~
dlor
+1000 on docker images being easier to construct than it seems.

I wrote a few blog posts awhile ago on this where I reimplemented docker pull
and push in bash: [https://www.danlorenc.com/posts/containers-
part-1/](https://www.danlorenc.com/posts/containers-part-1/)

It even got up to basic image modification support.

Disclosure: I work on kaniko and lots of other things that construct docker
images without docker at Google.

~~~
fulafel
This unpacks a series of tarballs fetched with curl, but it's not clear how it
would correctly handle file deletions.

~~~
errm
In the OCI image spec file deletions are handled with special whiteout
files... See: [https://github.com/opencontainers/image-
spec/blob/master/lay...](https://github.com/opencontainers/image-
spec/blob/master/layer.md#whiteouts)

------
gingerlime
We’re using docker for development, but we still have to take the leap into
production. The whole build/push/pull part is rather confusing somehow. I
tried docker hub or docker cloud build as it’s now called(?), but the build
itself takes forever... what are people using these days??

Also for development machines, how do you sync things between developers. I
can commit a docker file change, but unless I explicitly tell docker compose
to rebuild my images and containers, it will happily stick to the old version.
I have to keep nagging our (3) developers to do this from time to time... what
am I doing wrong?? Sorry if these are dumb questions but we’re still stuck
with the basics it seems.

~~~
lobster_johnson
If you're still struggling with the build workflow, it's probably not yet the
right time to take that leap.

It's not rocket science, of course. You _build_ an image somewhere (your local
machine, a CI server, anywhere), _push_ to a registry, and when you want run
the image, you _pull_ from the registry and run it. ("docker run" will, by
default, automatically pull when you ask it to run something.)

I don't quite understand what your Compose problem is. Is the Compose file
referencing images published to, say, Docker Hub? If so, the image obviously
has to be built and published beforehand. However, it's also possible to run
Compose against local checkouts, then run "docker-compose up --build", e.g.:

    
    
        version: '3.2'
        services:
          mainApp:
            build:
              context: .
          service1:
            build:
              context: ../service1
          service2:
            build:
              context: ../service2
    

and so on.

There's a whole ecosystem of tools built around Docker for building, testing,
deploying and orchestrating Docker applications. Kubernetes is one. If you're
having issues with the Docker basics, however, I wouldn't consider any of
these systems quite yet, although you should consider automating your building
and testing with a CI (continuous integration) system, rather than making your
devs build and test on their local machines.

As with anything, to actually use Docker in production you'll need an ops
person/team that knows how to run it. That could be something as simple as a
manual "docker run" or a manual "docker-compose", to something much more
complex such as Kubernetes. This is the complicated part.

~~~
gingerlime
The problem I was referring to with docker-compose:

let's say I update my Dockerfile and change from `FROM ruby:2.3.4` to `FROM
ruby:2.5.1` and commit the Dockerfile change, merge it to master, etc.

Our developers have to remember to manually run docker-compose --build, or to
remove their old containers and create new ones, which would get them
rebuilt... I couldn't find something that would warn them if they're running
off of stale images, or better, simply build them automatically when the
Dockerfile changes.

Part of the benefits of docker is creating a repeatable environment with all
sub-components on all dev machines. Isn't it?

Maybe our devs should only pull remote images and never build them, but then
wouldn't I have the same problem that docker-compose won't force or remind the
developers to pull unless they explicitly tell it to? And also, isn't this
detaching the development process around the Dockerfiles/builds themselves
from the rest of the dev process??

~~~
lobster_johnson
If you run with "docker-compose up --build", it should automatically build.
This requires that any app you want to work on references the local
Dockerfile, not a published one, the same way as in my paste. I.e. "build:
./myapp" or whatever.

Edit the code, then restart Compose, and repeat. It will build each time. If
you want to save time and you have some containers that don't change, you can
"pin" those containers to published images — e.g., the main app is in
"./myapp", but it depends on two apps "foo:08adcef" and "bar:eed2a94", which
don't get built every time. This speeds up development.

Building on every change sounds like a nightmare, though. It's more convenient
to use a file-watching system such as nodemon and map the whole app to a
volume. Here's a blog article about it that also shows how you'd use Compose
with multiple containers that use a local Dockerfile instead of a published
one: [https://medium.com/lucjuggery/docker-in-development-with-
nod...](https://medium.com/lucjuggery/docker-in-development-with-
nodemon-d500366e74df).

~~~
gingerlime
We're not building every time. But some times, like the example above, we do
need to build. The problem however is this becomes a fairly manual process. If
a developer forgets to do it, they will keep running with an older base image.
So all the consistency benefits across developers is gone.

In any case, thanks for your suggestions. I think it's some misconception on
my part about how docker-compose should behave.

~~~
atombender
So to me it's starting to sound like "developers forgetting" is your problem.
Not Docker or Compose.

The solution I've used in the multiple companies I've started is to maintain a
developer-oriented toolchain that encodes best practices. You tell the devs to
clone the toolchain locally and you build in a simple self-update system so it
always pulls the latest version. Then you provide a single tool (e.g.
"devtool"), with subcommands, for what you want to script.

For example, "devtool run" could run the app, calling "docker-compose --build"
behind the scenes. This ensures that they'll always build every time, and
never forget the flag.

If you have other common patterns that have multiple complicated steps or
require "standardized" behaviour, bake them into the tool: "devtool deploy",
"devtool create-site", "devtool lint", etc.

We've got tons of subcommands like this. One of the subcommands is
"preflight", which performs a bunch of checks to make sure that the local
development environment fulfills a bunch of checks (Docker version, Kubectl
version, whether Docker Registry auth works, SSH config, etc.), and fixes
issues (e.g. if the Google Cloud SDK isn't installed, it can install it). It's
a good pattern that also simplifies onboarding of new developers.

~~~
gingerlime
That's a great suggestion! Thanks. We're doing parts of it, but I just need to
expand it to work with docker-compose. As I mentioned, I probably had the
wrong preconceptions about it "figuring out" when components were stale... I
guess a few simple bash scripts can work wonders to make it more intelligent
:)

------
linkmotif
This is cool! Thanks for posting. I can see how this is useful if building
images is part of your CI process.

I’ve been using [https://github.com/dminkovsky/kube-cloud-
build](https://github.com/dminkovsky/kube-cloud-build) to build images on
Google Cloud Container Builder. It handles generating Cloud Container Builder
build requests based on the images specified in my Kubernetes manifests, which
was a big deal for me since writing build requests by hand was a total pain.

~~~
amq
If you have CI, you normally shouldn't need something like this.

~~~
rckrd
The idea is to not have to maintain separate CI infrastructure in addition to
your Kubernetes cluster.

Disclaimer: I worked on this kaniko at Google

~~~
CSDude
Thanks for your work really appreciated, but is there any way to cache some
layers?

~~~
dlor
There should be! We haven't thought too much about it yet, but there's no
reason it wouldn't work. We can't reuse the Docker daemon cache because that
would imply access to the docker daemon.

Discloser - I work on kaniko and other container things at Google.

------
spockz
My understanding is that it is best practice to run your docker builds and
images as s non root user. OpenShift will complain if you do for example. Now
this kaniko image runs the build as root contrary to the recommendation and
the post explicitly mentions this difference with Orca.

Why is it okay now for kaniko to run as root user?

~~~
humbleMouse
This was my first thought as well.

------
ofrzeta
With the availability of the free Red Hat tools for building container images
(buildah...) and this, it will be interesting to see what remains of Docker
(Inc).

~~~
zapita
It's pretty clear that Docker has been focused on moving downstream. They want
to add value by assembling open-source components into a complete platform
that they can control and sell. They _don 't_ want to be the ones developing
all the components themselves - at this level of maturity and sophistication
in the container market, they just don't have the manpower to do that. A major
benefit of that strategy is that they can use the best component available,
regardless of who developed it. I bet they're feeling spread very thin on the
open-source side, and would love to redirect some of their resources away from
developing a gazillion open-source gadgets on their own, and towards their
commercial products (which historically have been not as good in my
experience).

Evidence that Docker is doing this:

\- They only advertise three things with the name Docker: "Docker for Mac" (a
free product that is not open-source), "Docker EE" (an enterprise product),
and "Docker Hub" (a cloud service). Those are all downstream products, like
RHEL or Openshift.

\- The whole "Moby" thing is basically their upstream brand, aka "the things
not called Docker".

\- They spun out tons of smaller projects like buildkit, linuxkit, containerd,
runc, and seem eager to get others to use them and contribute, even
competitors.

\- They embraced Kubernetes as part of their downstream product, even though
they famously did not invent it, and they certainly don't control it.

So I think people saying "these free open-source tools are killing Docker" are
missing the point. The real competition for Docker is Openshift vs Docker EE,
everything else is implementation details.

If you listen to the sales pitch of these two companies right now, it's an
absolute tug of war. Docker focuses on independence and innovation ("we know
where containers are going, and we don't force RHEL down your throat"). Red
Hat focuses on maturity and upstream control ("We've been by your side for 20
years, are you going to trust us or some Silicon Valley hipster? Also we
employ more Kubernetes contributors than anyone else").

That's the real battle, in my experience on the open-source side you'll find
mostly engineers from all side collaborating peacefully and building whatever
they need to get their job done.

------
bkeroack
Interesting! We built and use a service called Furan:
[https://github.com/dollarshaveclub/furan](https://github.com/dollarshaveclub/furan)

That said, Furan isn't suitable for untrusted Dockerfiles (or multi-tenant
environments) exactly due to the security implications of access to the Docker
engine socket.

The issue I see with Kaniko is drift from upstream Moby/Docker syntax. One of
the strengths with Furan is that you have the guarantee that the Docker build
you perform locally is exactly what happens by the service. When you can't
make this guarantee you get into weird situations where "the build works for
me locally" but there's some issue when doing a remote build. That's also why
we've resisted putting special build magic into Furan (like injecting metadata
into the build context, for example).

------
zapita
Does this (or could this) use Buildkit? It seems that Docker themselves are
encouraging the development of an ecosystem of third-party container build
tools, with buildkit as an interoperability layer. I heard good things about
buildkit but haven't tried it yet.

If Kaniko authors are reading this: have you considered buildkit and, if not,
would you be open to contributions based on it?

My understanding is that the official 'docker build' itself is based on
Buildkit.

[https://github.com/moby/buildkit](https://github.com/moby/buildkit)

~~~
dlor
Kaniko doesn't use buildkit - buildkit still uses containerd/runC under the
hood so it can't run inside a container easily.

We are looking at interoperability with buildkit (and the large set of of
other tooling like this) through the CBI:
[https://github.com/containerbuilding/cbi](https://github.com/containerbuilding/cbi)
which aims to be a neutral interface on top of things like buildkit, buildah,
docker and kaniko that build images.

Discloser: I work on kaniko and other container things at Google.

~~~
zapita
Very interesting, and it looks like the people working on CBI are also active
on Buildkit, which is a good sign!

Thank you for the pointer.

~~~
AkihiroSuda
@zapita

FYI: Kaniko plugin for CBI is now available.
[https://github.com/containerbuilding/cbi/pull/35](https://github.com/containerbuilding/cbi/pull/35)

~~~
zapita
Thanks Akihiro for the follow-up. And while I'm at it, thank you for all this
excellent code that we get to enjoy for free! It's really fantastic work.

------
tyrankh
This is great work. Github link for the lazy:
[https://github.com/GoogleCloudPlatform/kaniko](https://github.com/GoogleCloudPlatform/kaniko)

------
ksajadi
This is a great tool. Wish it could work with build workflow tools like
Habitus ([http://www.habitus.io](http://www.habitus.io))

------
medyadaily
Sweet, This is truly great, I was hoping for a service like this for a long
time. being able to build images without root privilege!

~~~
amouat
Kaniko does run in the container as root, but the container doesn't need to be
granted any extra privileges when run (you don't need the equivalent of
Docker's --privileged flag).

