

Always ship trunk: Managing change in complex websites  - brown9-2
http://www.paulhammond.org/2010/06/trunk/alwaysshiptrunk.pdf

======
bdb
For what it's worth, this is pretty much exactly what we did at the last web
property I ran with ~500 million monthly pageviews. We had a configuration
framework that allowed us to flip bits on the fly (and, importantly, flip them
back), without a deploy, in ~1 second across all of our app servers. The app
servers listened to a message queue which would broadcast changes of
configuration values.

This one thing alone made a huge difference in our engineering velocity.

If you're looking for a pre-built component to do these things for you today,
check out Apache Zookeeper: <http://hadoop.apache.org/zookeeper/>

~~~
shykes
Yes. If you're in this situation Zookeeper will change your life.

------
swombat
Good points, and we're doing almost exactly that, with one difference.

I think it's worth having 2 branches you ship from: master and production (or,
in reverse, master and development).

The reason is that this way, if you're in the middle of some relatively large
and untested changes in the development branch, you still have a copy of
exactly what's on production to work on in case there's an urgent patch that
needs to go out.

This is not strictly necessary, of course, but we've found that it's saved us
some hassles.

------
thyrsus
At the end, he asks "What would a version control system for a web app look
like?" Perhaps that's only a rhetorical question, but I think he's already
touched on a partial answer: it's a configuration management system. A good
configuration management system will help you keep track of what's currently
deployed where (although variation per login id is generally out of scope) and
help you be sure that the requirements for deployment are met. The source for
the configuration is normally kept under revision control for non-branch
elements that revision control does well: _why_ the change went in, _who_ put
the change it (in case the "why" was incomplete ;-}), _what_ the change was
(centralized diff), and _when_ the change went in. _When_ can be important for
those little used but essential features that you find out stopped working
some time ago, and you want to find out _what_ made it break.

He advises discipline in removing unused configurations. I'd like to hear what
sort of dependency analysis tools support that activity.

~~~
lars512
Doesn't Google App Engine provide some of this for free? You upload versions,
change which one is deployed, and can easily roll back at a moment's notice.
It's not as swift or sexy as multiple configurations in the same code, but
still far nicer than managing it yourself.

------
patio11
This will seem a lot less heretical if you do a lot of A/B tests, since if
you're already supporting 2^13 versions of the code running simultaneously,
heck, what is a couple more.

------
macrael
It's easy to miss, but this talk is given by a Flickr engineer and is how they
deploy their code. Pretty big recommendation.

~~~
neilk
Former eng manager actually.

In this preso he's focusing in on just one aspect of how Flickr does things.
He did a more wide-ranging talk with ops guru John Allspaw last year.

<http://velocityconference.blip.tv/file/2284377>

For instance, as people noted already, he's advocating testing features on
live servers -- in a controlled way. This doesn't work unless you have the
kind of dev/ops cooperation that he describes in the above video, as well as
practices for quickly rolling back changes and a culture of not blaming
people.

------
T_S_
Wow, I just learned a new idea. Test SaaS on production servers. Sounds scary
but makes sense. Fast driving is not always dangerous driving. Great post.

------
JoachimSchipper
Sounds like a bit of tool support would be useful.

Note that the UNIX command patch has a -D <symbol> option, which results in
the patch only being applied if the (C/C++) program is compiled with -D
<symbol>, so this idea is not actually all that new...

------
Maro
Good 5 minute read.

Last slide asks a great question:

 _What would a revision control system built for supporting deployed web
applications be like?_

------
jpcx01
After much experimentation over the years, I've found the best approach as
follows (using git):

\- always ship master

\- always develop using topic branches (branches dedicated to a single
feature)

\- deploy topic branches to dev/staging (with large visible footer label text)

\- don't leave old topic branches around on your central git server (delete as
soon as its tested and merged back into master)

------
stcredzero
My first thought is that this should be managed using a Factory pattern. If
one is truly doing A/B testing, then this is manageable for N = 2, and a lot
of repeated conditional code with the same if-then-else statement will
disappear.

If they are A/B testing a lot of things, then this can get unmanageable. For
that, how about a collection of functions? (methods? I am not trying to be
language-specific at all here.) Have a facade which stores references to the
actual functions. This is much easier if your application is structured in a
functional or nearly-functional way.

Writing 100 if-then-else statements without making a single error is doable.
But making the conditional logic disappear and just having to manage the
methods themselves and a configuration is much easier.

~~~
iamwil
That was my thought at first as well, but probably for illustrative purposes,
it was easier to do if-else statements. The more structure there is, the
higher the learning curve. For a presentation, it probably wasn't necessary.

~~~
nostrademons
In practice, it turns out to be easier to do with if-else statements as well.

A factory pattern is useful when you know ahead of time along what dimensions
your system will change. You can build an interface with methods for each of
those change points, then call out to those methods when appropriate. For each
experimental variant, you provide a different implementation and instantiate
it based on experiment selection at the factory.

Unfortunately, these types of configuration changes, by definition, tend to be
_experimental_. Meaning you have no idea how the app is going to change to
accommodate them, you just have an idea that might or might not help users and
you want to test that hypothesis as quickly as possible. Rearchitecting your
app to build the appropriate factory interfaces takes significantly longer
than just throwing in an if, and it's all wasted effort if the experiment
comes back with a null result (as many do).

If you find yourself making experimental changes to the same areas often,
_then_ you might want to look at introducing a Factory. But design patterns in
general are something you reach for once the system has matured, not something
you do when you're still experimenting and exploring the domain.

~~~
stcredzero
If you read my post carefully, I propose a Factory, then reject it in favor of
something simpler.

My proposal reduces the refactoring to "this block of code becomes a function
call." There's still a piece of conditional logic, for each dimension, but it
only gets written once. The application code gets smaller by applying this,
except for one startup routine where we set up the thunk. Overall, it's
actually less code than sprinkling conditionals everywhere. You could set
things up so you change the configuration of all your A/B experiments in a
single file. It's not so much a pattern as it is a reorganization of the logic
along a straightforward application of DRY.

One might still want to stick with the conditionals everywhere if most of the
code needs local state. This is probably an indication of a need to refactor,
though.

------
dedward
Thank you - very concise and will help me personally in a current initiative
to drive some points home!

~~~
dedward
I suppose some details would be appropriate....

I'm dealing with a suite of what used to be plain-old websites that slowly,
over time, started to gain functionality. PHP crept in there (just for
includes) - the web team went from one person to an entire department in a
short period of time, and some functionality keeps creeping in there - some of
it done by developers, some of it done by web PHP guys who shouldn't really be
writing code. Incremental direct updates to the site on a daily basis are a
regular thing by far too many people.

I'm sure many people have the same story.

The challenge is in taking the current situation and turning it into something
manageable with release control, testing, and so on - because it's at a point
where a simple mistake by a designer can (and has) effectively taken the site
down, and due to the nature of the update process, it's hard to find out 'wtf'
just happened. The GOOD part is everyone is on-board with changing things (no
politics).

The hard part is figuring out what those changes are - as soon as we started
trying to fit this into the standard developer workflow, it was obvious that
we were adding far, far too much un-needed technical bureaucracy (do we have a
word for that?) for no real benefit - and the idea that the release and
deployment tools are where the real benefit is is really becoming apparent -
the version-control system is necessary, but it's not strictly the most
important piece.

So for me, these slides (wish I could see the talk that went with it) really
helps cement that I'm not insane for thinking differently about how this
should work.

------
10ren
btw: scribd works really well now.

------
metachris
92 slides is a bit long for the points the author is making, although it's
hard to say without hearing the talk.

Basically the author prefers GIT and mercurial over subversion because of
easier branching, and argues for having multiple versions of the backend
running in order to switch between them with with configs in your code (eg.
cfg["use_new_awesome_backend"] = 1) in multiple variations (eg. if
(rand(0,99)<10) cfg[".."]=1, etc).

~~~
carbocation
>Basically the author prefers GIT and mercurial over subversion because of
easier branching

That wasn't exactly the conclusion I got out of it. I felt like he expressed a
slight preference for git and mercurial, but at the end he asked for people to
envision what a web/SAAS version control system should really look like, since
none of the 3 fit that bill, in his opinion.

(I agree with the second half of what you wrote that I'm not quoting here.)

~~~
metachris
True, I missed the point in the first part. Thanks for pointing that out.

