This one thing alone made a huge difference in our engineering velocity.
If you're looking for a pre-built component to do these things for you today, check out Apache Zookeeper: http://hadoop.apache.org/zookeeper/
I think it's worth having 2 branches you ship from: master and production (or, in reverse, master and development).
The reason is that this way, if you're in the middle of some relatively large and untested changes in the development branch, you still have a copy of exactly what's on production to work on in case there's an urgent patch that needs to go out.
This is not strictly necessary, of course, but we've found that it's saved us some hassles.
He advises discipline in removing unused configurations. I'd like to hear what sort of dependency analysis tools support that activity.
In this preso he's focusing in on just one aspect of how Flickr does things. He did a more wide-ranging talk with ops guru John Allspaw last year.
For instance, as people noted already, he's advocating testing features on live servers -- in a controlled way. This doesn't work unless you have the kind of dev/ops cooperation that he describes in the above video, as well as practices for quickly rolling back changes and a culture of not blaming people.
Note that the UNIX command patch has a -D <symbol> option, which results in the patch only being applied if the (C/C++) program is compiled with -D <symbol>, so this idea is not actually all that new...
- always ship master
- always develop using topic branches (branches dedicated to a single feature)
- deploy topic branches to dev/staging (with large visible footer label text)
- don't leave old topic branches around on your central git server (delete as soon as its tested and merged back into master)
Last slide asks a great question:
What would a revision control system built for
supporting deployed web applications be like?
If they are A/B testing a lot of things, then this can get unmanageable. For that, how about a collection of functions? (methods? I am not trying to be language-specific at all here.) Have a facade which stores references to the actual functions. This is much easier if your application is structured in a functional or nearly-functional way.
Writing 100 if-then-else statements without making a single error is doable. But making the conditional logic disappear and just having to manage the methods themselves and a configuration is much easier.
A factory pattern is useful when you know ahead of time along what dimensions your system will change. You can build an interface with methods for each of those change points, then call out to those methods when appropriate. For each experimental variant, you provide a different implementation and instantiate it based on experiment selection at the factory.
Unfortunately, these types of configuration changes, by definition, tend to be experimental. Meaning you have no idea how the app is going to change to accommodate them, you just have an idea that might or might not help users and you want to test that hypothesis as quickly as possible. Rearchitecting your app to build the appropriate factory interfaces takes significantly longer than just throwing in an if, and it's all wasted effort if the experiment comes back with a null result (as many do).
If you find yourself making experimental changes to the same areas often, then you might want to look at introducing a Factory. But design patterns in general are something you reach for once the system has matured, not something you do when you're still experimenting and exploring the domain.
My proposal reduces the refactoring to "this block of code becomes a function call." There's still a piece of conditional logic, for each dimension, but it only gets written once. The application code gets smaller by applying this, except for one startup routine where we set up the thunk. Overall, it's actually less code than sprinkling conditionals everywhere. You could set things up so you change the configuration of all your A/B experiments in a single file. It's not so much a pattern as it is a reorganization of the logic along a straightforward application of DRY.
One might still want to stick with the conditionals everywhere if most of the code needs local state. This is probably an indication of a need to refactor, though.
I'm dealing with a suite of what used to be plain-old websites that slowly, over time, started to gain functionality. PHP crept in there (just for includes) - the web team went from one person to an entire department in a short period of time, and some functionality keeps creeping in there - some of it done by developers, some of it done by web PHP guys who shouldn't really be writing code. Incremental direct updates to the site on a daily basis are a regular thing by far too many people.
I'm sure many people have the same story.
The challenge is in taking the current situation and turning it into something manageable with release control, testing, and so on - because it's at a point where a simple mistake by a designer can (and has) effectively taken the site down, and due to the nature of the update process, it's hard to find out 'wtf' just happened.
The GOOD part is everyone is on-board with changing things (no politics).
The hard part is figuring out what those changes are - as soon as we started trying to fit this into the standard developer workflow, it was obvious that we were adding far, far too much un-needed technical bureaucracy (do we have a word for that?) for no real benefit - and the idea that the release and deployment tools are where the real benefit is is really becoming apparent - the version-control system is necessary, but it's not strictly the most important piece.
So for me, these slides (wish I could see the talk that went with it) really helps cement that I'm not insane for thinking differently about how this should work.
Basically the author prefers GIT and mercurial over subversion because of easier branching, and argues for having multiple versions of the backend running in order to switch between them with with configs in your code (eg. cfg["use_new_awesome_backend"] = 1) in multiple variations (eg. if (rand(0,99)<10) cfg[".."]=1, etc).
That wasn't exactly the conclusion I got out of it. I felt like he expressed a slight preference for git and mercurial, but at the end he asked for people to envision what a web/SAAS version control system should really look like, since none of the 3 fit that bill, in his opinion.
(I agree with the second half of what you wrote that I'm not quoting here.)
See also: Lawrence Lessig's presentation on Free Culture (http://randomfoo.net/oscon/2002/lessig/free.html) That's probably 1000 slides, but they are used expertly to deliver a clear, concise message.