

Deployment Infrastructure for Continuous Deployment at kaChing - dkf
http://eng.kaching.com/2010/05/deployment-infrastructure-for.html

======
russell_h
Very interesting. How often do problems get past your testing? Do you do any
sort of automated UI testing, or is this only for backend code?

~~~
dkf
Fewer and fewer. As soon as an issue gets by our testing, we get the person
who shipped the bug, and a few of our testing gurus together to figure out how
to add a test for it. We've built some really impressive testing
infrastructure, which we're working to open source. If I was forced to put a
number on it, we may see a small issue every other day, and a larger one
roughly once a week.

As soon as we rolled out the Deployment Manager we saw problems staying on
production for shorter and shorter times. We see the DM as more issue
mitigation than testing. The self test production framework we're continuing
to build out has been hugely helpful.

The blog post is mostly about backend code. At the moment we're using selenium
in more of a monitoring role (think keynote but free). We need to get selenium
up and running as part of frontend automated testing, but right now we're
pretty reliant on Test::Unit and some basic integration tests.

------
locopati
Do your services use SQL databases? If so, how do you coordinate service code
deployment that also requires database changes?

~~~
dkf
Yes, where we feel it's the best solution (our standard key-value store is
voldemort). SQL changes are peer reviewed and applied with a set of scripts we
have to make the process easier. This isn't integrated into the deployment
service yet, but it's something we've talked about.

<http://news.ycombinator.com/item?id=411282> has a good description of
something that's very similar to our SQL rollout policy:

"only add columns to tables and new tables, never to remove anything, so the
database can be upgraded hot, and the old code can continue to run on it, only
the new code will see any new columns/tables."

and

"Once you are sure you are never going to roll back to the old code you can
wipe out the old columns."

This comment (and its author) <http://news.ycombinator.com/item?id=411358>
were formative in my views of service deployment.

