

Divide and Concur - testing code at etsy - dajobe
http://codeascraft.etsy.com/2011/04/20/divide-and-concur/

======
bestes
Sounds like you've developed a great system/process here. Are you a developer
or QA? Are there two groups or even just a person with testing as their
responsibility?

Have you ever had people mark tests @flaky just so they could deploy (and that
later caused grief)?

I would love to hear more.

~~~
NoahSussman
Both of the authors are Web developers with a deep interest in software
quality and automated testing. Etsy has followed the same software quality
model as Facebook: a dedicated testing team, but no dedicated manual QA team.
See also <http://b.qr.ae/hJVb4W>

Etsy's engineering culture puts great emphasis on personal responsibility and
taking pride in the quality of one's work. Because of that, no, there has
never yet been a problem with people taking tests offline simply so they could
deploy. The issue is more that tests "wear out" over time and need to be
either rewritten, replaced or discarded (for more on this phenomenon see Boris
Beizer and his "pesticide paradox").

------
draz
sounds cool, though, with 25 deployments a day, it seems like you're just
deploying quick fixes. How involved could changes be with such small time
intervals? I applaud the approach though. Also, could you also talk about how
you track all the end-user impact, given the very frequent changes? How does
this feed back into the analytics (is change A causing rise/fall in
conversion, or change B)?

~~~
NoahSussman
Yes, most deployments contain only a couple of changesets -- ideally no more
than a few dozen lines of code per push. And even these small changes are
deployed behind config flags. Flickr has a similar continuous deploy strategy,
see <http://bit.ly/esoXiX>

There is intended to be no end user impact whatsoever. Releasing behind config
flags means that feature rollouts are decoupled from deploying code. New code
runs "dark" for a long time in production. There we can log and graph and
adjust its performance before it is ever visible to the user.

And of course, _we_ can see dark features in production before they're
released or even complete. This means that at Etsy we've been using and
tweaking our features long before they are made public.

Since we've been closely monitoring and recording the performance of each new
feature since its inception, we are highly aware of how that changes once the
code is live to a known, small number of our users. Gather data, make
adjustments, rinse and repeat until eventually the feature is live to 100% of
users.

See also

[http://codeascraft.etsy.com/2011/02/04/how-does-etsy-
manage-...](http://codeascraft.etsy.com/2011/02/04/how-does-etsy-manage-
development-and-operations/)

<http://codeascraft.etsy.com/2010/12/08/track-every-release/>

[http://codeascraft.etsy.com/2011/02/15/measure-anything-
meas...](http://codeascraft.etsy.com/2011/02/15/measure-anything-measure-
everything/)

