
Container Structure Tests: Unit Tests for Docker Images - mzl
http://opensource.googleblog.com/2018/01/container-structure-tests-unit-tests.html
======
ukoki
My standard way of "smoke testing" docker images is to run a verify script at
the end that just checks all the expected binaries are in the path, and tries
to access any expected environment variables while `-u` is set.

[https://github.com/EngineerBetter/concourse-
up/blob/master/c...](https://github.com/EngineerBetter/concourse-
up/blob/master/ci/docker/Dockerfile#L49-L50)

[https://github.com/EngineerBetter/concourse-
up/blob/master/c...](https://github.com/EngineerBetter/concourse-
up/blob/master/ci/docker/verify_image.sh)

~~~
dlor
Maintainer of container-structure-test here.

Cool example! We started with something like this, but ran into some issues
when trying to test images that don't contain a shell.

------
crdoconnor
This feels to me like the wrong approach. You shouldn't be trying to test if a
file exists or a command was run as a 'unit test' (as a sanity check, maybe,
but not a test) - that's testing the implementation.

You should be testing for the _desired behavior_ that having that command run
or having that file there was supposed to achieve.

IMHO even the use of containers should probably be considered an
implementation detail.

~~~
puzzle
Sometimes the desired behaviour is actually generating a Docker image with
specific contents. Think of the pause container image that every Kubernetes
cluster out there uses. To put it in other way, this is for build targets
producing Docker images that are needed by an arbitrary set of projects,
possibly from third parties.

Suppose you have an organization-wide Docker image from which tens of other
images from a number of teams are derived. You could add a test to make sure
that the ca.crt file is always packaged and never empty or you could let tens
of downstream tests and images fail in mysterious ways when it's missing
(unless you replicate the check in all projects depending on the base image).

The use case is more for infrastructure that explicitly deals with or supports
containers. Generic applications, as you say, shouldn't bother.

~~~
crdoconnor
>Think of the pause container image

The pause container is very much an implementation detail. That's sort of my
point. Why would you try to test implementation details?

>You could add a test to make sure that the ca.crt file is always packaged

I could, but why? If I wrote a command to put ca.crt there I would expect it
to be there unless somebody deliberately took it out.

It would be even better if I wrote a command to put ca.crt in the wrong place
and then tested to see that it was in the wrong place. The test would be worse
than useless in that case - it would fail on a working container and pass on a
broken container.

The ca.crt is there because it triggers desired behavior - e.g. that you can
connect to services over SSL without errors. Test _that_ instead.

>The use case is more for infrastructure that explicitly deals with or
supports containers.

Personally I'd consider containers themselves an implementation detail most of
the time - the exception being when you're building containers for other
people to use.

I still think it's the wrong approach for that too.

~~~
dlor
> The ca.crt is there because it triggers desired behavior - e.g. that you can
> connect to services over SSL without errors. Test that instead.

Awesome example here! We have a test that uses container-structure-test to do
exactly this over here:
[https://github.com/GoogleCloudPlatform/distroless/blob/1ea91...](https://github.com/GoogleCloudPlatform/distroless/blob/1ea91a52f18cf99cc3f71f55365f6888bb414b05/java/testdata/certs.yaml)

You can mix and match between tests that check file contents and tests that
actually run commands inside the container and make assertions based on their
output.

~~~
crdoconnor
Don't you think that your test for the presence of the cacerts file in
java.yaml is kind of redundant in that case?

------
v_lisivka
Just package content into RPM and use rpm to perform these checks at
installation time as usual. Moreover, Dockerfile will be simple "yum install
package" command.

------
willejs
This could be quite useful to define the basic things a container should
exhibit. A lot of the time you rely on certain files being present or a
specific entrypoint and command combo when running containers under
orchestration with sidecars or volume mapping etc. So this could help you
define what is needed and why for others who will change your container build
steps in future.

It could also compliment your Goss or Inspec integration tests quite nicely.

------
mooreds
That's awesome. One step closer to treating infrastructure exactly as code.

I'd imagine you could stop a build if the docker image generated doesn't have,
say, a valid python installation because someone mistyped a command, and that
doing so would be quicker than standing up the image and running an external
test against it.

~~~
dboreham
To me this sounds crazy. The possibilities from "mistyped commands" are
essentially infinite.

Instead, test that the container does what it is supposed to do, at least in
some minimal way.

Then if it doesn't do what it should do because there's no Python, it won't do
what it should do and the test will fail.

One test to catch the infinite number of typing errors.

