
Testing with Jenkins, Ansible and Docker - cpsaltis
http://blog.mist.io/post/82383668190/move-fast-and-dont-break-things-testing-with
======
SeoxyS
You should optimize your RUN commands. Every time you RUN in a Dockerfile, it
creates a new filesystem layer. There's a hard limit (42, iirc) to the number
of layers that Docker can support.

Instead of doing:

    
    
        RUN echo bar >> foo
        RUN echo baz >> foo
    

You could do:

    
    
        RUN echo bar >> foo \
            echo baz >> foo

~~~
1331
In my humble opinion, each layer should have a clear purpose:
[http://www.extellisys.com/articles/docker-layer-
overload](http://www.extellisys.com/articles/docker-layer-overload)

~~~
shykes
You make good points in this post. We are working on addressing some of them
already. I would love to discuss this directly, you can ping me anytime,
shykes on #docker / freenode.

------
Silhouette
_At the same time, we need to move fast with development and deliver updates
as soon as possible. We want to be able to easily deploy several times per
day._

Genuine question, not intended as any sort of troll: what benefits would
people with this philosophy say their organisation gains from routinely
deploying multiple times per day?

I have nothing against better testing tools or more efficient development
processes, of course, and if you have a serious bug then being able to fix it
as quickly as possible is obviously beneficial. I just don't understand where
this recent emphasis on _always_ trying to move fast has come from, or what
kind of management strategy someone might use to take advantage of such
agility.

~~~
josh-wrale
I found this article helpful in understanding the 'multiple times per day'
motive.

[http://martinfowler.com/articles/continuousIntegration.html](http://martinfowler.com/articles/continuousIntegration.html)

~~~
Silhouette
The motivation for continuous _integration_ I can certainly understand. I'm
sure many of us have worked on projects where someone was working on a branch
for an extended period to implement some big new feature, and then ran into
the mother of all merge conflicts when they finally wanted to release their
changes for general use. I don't think CI is appropriate in all cases, but I
find it a better strategy than mega-merges for most projects, most of the
time.

It's specifically the motivation for continuous _deployment_ that I'm curious
about here. Fixing bugs quickly is obviously a plus, and I found the point
'twic mentioned about rapidly instrumenting a production system interesting.
These are good arguments for being able to release an update on demand, and
indeed for continuous integration as a technique to support that goal.

However, to me neither seems like a strong argument for routinely releasing
multiple updates per day. I've always been a little surprised that so many web
sites and SaaS applications seem to take pride in making such frequent
changes, even though doing so surely incurs some risk of breaking things and
typically also causes some degree of instability in the UI. I was just
wondering whether anyone had experiences and/or actual data they could share
about whether the practice made a measurable improvement to, for example,
revenues or reported customer satisfaction -- something concrete that might
outweigh the risks -- or whether it's more of a subjective preference for that
way of working.

~~~
gtaylor
> However, to me neither seems like a strong argument for routinely releasing
> multiple updates per day.

I don't understand this line of thinking. If there's a bug that you can fix in
short order, why not fix it and deploy said fix? If you've set your CI and
infrastructure up correctly, update rollouts aren't something the user even
notices.

I've always been a big proponent of small, atomic commits. Being able to
deploy just one thing at a time means we often know where breakage comes from
(when it happens), and we can respond to feedback sometimes within minutes.

The customers love it, and we enjoy excellent stability and development
velocity. _These_ are the reasons we deploy many times a day. There's no
reason for us _not_ to deploy things once they are tested and ready.

~~~
Silhouette
_If there 's a bug that you can fix in short order, why not fix it and deploy
said fix?_

If you've got an important bug to fix, sure, of course you want it dealt with
as quickly as possible.

 _If you 've set your CI and infrastructure up correctly, update rollouts
aren't something the user even notices._

Not if it's for a bug fix or security patch, but presumably most of your
development work is adding new features? In that case, isn't the whole point
that it's something the user will notice?

 _I 've always been a big proponent of small, atomic commits._

And again, no argument from me there. While I don't think every development
project is suitable for that approach, I favour it most of the time myself.

 _Being able to deploy just one thing at a time means we often know where
breakage comes from (when it happens)_

This is where my experience and background seem to be very different to the
deploy-hourly kind of philosophy. I've noticed that proponents of the latter
often seem to assume that stuff is going to be breaking all the time, and
therefore that being able to deploy bug fixes or diagnostics very quickly will
be a significant advantage.

However, if you have so many significant bugs that you need to deploy multiple
times per day just to keep up, to me that seems like a clear demonstration of
insufficient QC/QA in the development process. And so I can't help wondering
whether the pressure to release so often, and the consequent almost exclusive
reliance on some sort of automated test suite for quality checking, is a
causal factor in having so many bugs in the first place.

 _The customers love it, and we enjoy excellent stability and development
velocity. These are the reasons we deploy many times a day. There 's no reason
for us not to deploy things once they are tested and ready._

I guess this is the paradoxical part that I don't understand. If your project
enjoys excellent stability and your deployments are all soundly tested before
they go live, then by definition you don't need the rapid deployments just to
rush out bug fixes or diagnostics.

So that brings me back to where I came in: do you experience any concrete,
measurable benefits from the "excellent development velocity" you mentioned,
or is it simply your team's preferred development style?

~~~
gtaylor
> So that brings me back to where I came in: do you experience any concrete,
> measurable benefits from the "excellent development velocity" you mentioned,
> or is it simply your team's preferred development style?

Yes, see the following:

> The customers love it, and we enjoy excellent stability and development
> velocity. These are the reasons we deploy many times a day. There's no
> reason for us not to deploy things once they are tested and ready.

We're not just deploying bug fixes quickly (which you seem to be fixating on a
bit), we are constantly making small iterations. If we're handling support and
see something that could be improved, we improve it and roll it out quickly.
If we have a moderate sized feature branch, it is merged, tested, and rolled
out immediately as soon as it passes QA. No need to set a rigid "We only
deploy on Wednesdays" schedule. Pass QA, deploy immediately.

> Not if it's for a bug fix or security patch, but presumably most of your
> development work is adding new features? In that case, isn't the whole point
> that it's something the user will notice?

No, a huge chunk of dev time is spent iterating on and improving existing
features. There gets to be a point in a product's life cycle where you are
more or less feature complete, where the emphasis shifts to refining and
improving what you already have. Sure, there are new features, but if you
never iterate, you've got older stuff accumulating dust and ageing badly.

------
wiremine
I've been using Ansible for a few production-related tasks lately, and think
it's great. It provides the right level of abstraction, IMHO: you can crack
open a playbook, read through it, and know exactly what it is doing. There's
also a growing number of playbooks if you google around.

That said, the biggest downside I've seen with Ansible is reusable components.
They have something called Galaxy in beta [1], which should help, although it
feels a bit rough yet...

[1] [https://galaxy.ansible.com/](https://galaxy.ansible.com/)

~~~
crdoconnor
>That said, the biggest downside I've seen with Ansible is reusable components

This was a major, major issue with puppet, too - although somewhat worse, in
that the manifests were more unreadable.

One issue that ansible still hasn't really solved is the horrible
if/then/elses you have to put in to accommodate RHEL/ubuntu/centos/whatever
else differences.

I like that galaxy at least lets you specify which environment the playbook
will run on, though.

~~~
mpdehaan2
Check out the group_by module, as this can make those totally go away.

------
trjordan
Cool stuff! We actually use a similar setup [1], but with the additional
hassle of figuring out how to handle hardware connections back to the app.
Docker + Jenkins has definitely been a big win.

[1] [http://www.appneta.com/blog/automated-testing-with-
docker/](http://www.appneta.com/blog/automated-testing-with-docker/)

------
regecks
Take a look at github.com/drone/drone for an upcoming CI platform built around
Docker. It's totally open source, but you can use the hosted version at
drone.io as well.

