
Feature Toggles - okfine
http://martinfowler.com/articles/feature-toggles.html
======
xorcist
Architecture people tend to speak in trivial examples. Swapping out an
algorithm with a backwards compatible one? Good for you! It doesn't matter how
you do it because it's the most trivial example one could think of. (A more
realistic example would be new functionality that touches large parts of the
code base, or compliance with a new API that the payment processor switches
that has more data on every payment as it crosses our systems.)

I never understood why some people are scared of branching. It seems to work
pretty well for Linux development. Is your project really that much more
complex? As long as you merge continously from upstream, you're golden. What's
the problem you're trying to solve, exactly?

I _do_ have some reservations against #ifdef-sprinkled code. And it's not an
unlikely scenario that your feature toggles turn into something very similar,
if conditions change and maybe they're not that temporary anymore.

~~~
mikekchar
I'm not quite sure I understand all of your comment, but I can respond to why
one might want feature toggles rather than branches.

One of the activities that can really improve the quality of code on a team is
continuous integration. Now, I don't mean setting up a "CI" server to run
tests for you when you push your code. I mean _actually_ merging your code
with your fellow programmers every 20 minutes or so.

The reason this is such a powerful technique is that it gets people to see
what you are doing pretty much as soon as you do it. If you do something
stupid (as we all do), they complain instantly because they will merge in your
code every 20 minutes. If you do CI well, there are no lingering surprises or
hard decisions of "Well, maybe you should start all over again".

The problem is that all the code is mixed up. This is completely fine if you
have 1 or 2 week sprints and deploy at the end of the sprint. The problem is
that this kills another really power technique called "continuous deployment".
Ideally, we would like to deploy as soon as a story is finished. But if it's
mixed up with a whole bunch of other code, then you need to make sure that the
other code isn't going to break something. Enter feature toggles.

It's a challenge to support both CI and CD at the same time. You can never
break any code. It takes a lot of discipline, but if you can manage it, I
guarantee pretty enormous improvements in both productivity and code quality
(and even programmer happiness). And before you ask, yes I have had teams
which did CI well, but no, we didn't do CD.

It's not for everyone. But as I said, it's pretty powerful if you manage it.

~~~
onion2k
_You can never break any code._

That's a ludicrous expectation will lead to developers being scared of merging
changings, which will in turn lead to them hiding mistakes, refusing to talk
through their code, and ultimately hacking things in to the codebase that
"work" but are terrible solutions to problems.

It's _far_ better to have a safe environment where a dev can branch a new
feature, break the whole codebase _completely_ if necessary, and only merge
things back in when they're confident that their solution is a good one (with
a review of the pull request by a maintainer or two).

The aim of writing software is to make a good product that solves a problem.
It is not to 'never break the build' or 'merge as often as possible'.

~~~
lmm
> That's a ludicrous expectation will lead to developers being scared of
> merging changings, which will in turn lead to them hiding mistakes, refusing
> to talk through their code, and ultimately hacking things in to the codebase
> that "work" but are terrible solutions to problems.

Maybe, but it can just as easily lead to good practice. E.g. if you're scared
to change a particular piece of code, that's a red flag that that particular
piece of code is inadequately tested - and "am I scared?" is a much better
metric than automated test coverage reports.

------
gioele
I find the `scientist` gem by GitHub [1] a very nice way to approach the
problem of testing alternative implementations.

Basically `scientist` always runs both the original code and the new code.

    
    
      def allows?(user)
        science "widget-permissions" do |e|
          e.use { model.check_user(user).valid? } # old way
          e.try { user.can?(:read, model) } # new way
        end # returns the control value
      end
    

It returns the result given by the original code, but it also compares both
results and store statistics about when the two codes produce different
results. This allows you to run experiments directly in the deployed branch
without any service disruption.

[1] [https://github.com/github/scientist](https://github.com/github/scientist)

------
geocar
A more powerful version of these things are called dynamic variables, and
they're worth some study beyond feature selection and testing: Otherwise you
just get global variables with funny names.

Many languages have some dynamic variables (this, self), and few languages
have first-class support for them (notably CL, Perl). While dynamic variables
are tricky to emulate fully, as long as you have the ability to catch
exceptions and closures, a very fair simulation can be made:

    
    
        (function(){
          var o={};
          D=function(k){return o[k]}
          dlet=function(b,f){
            var r,s={};function oops(){for(var k in s)o[k]=s[k]};
            for(var k in b)s[k]=o[k],o[k]=b[k];
            try{r=f()}catch(e){oops();throw e;}; oops();return r;
          };
        })();
    

Besides feature variables that are globally accessible, one useful thing you
can do with dynamic variables is set up _error handlers_. Consider the
situation where you are saving a big file to the disk and run out of space. If
you:

    
    
        save_stuff(); if(oops)throw 'out of space'; save_more_stuff();
    

then your slow saving operation needs to be retried from the start, however if
you use a dynamic variable to look up the handler, you can:

    
    
        save_stuff(); if(oops)D('out of space')(next); else next();
        function next(){ save_more_stuff(); }
    

The user can then use their fancy multitasking environment to clean up some
space, and we can continue our operation. I generally recommend this form of
error handling anyway.

Another useful thing is for context: Imagine you have a user interface choice
between an HTTP response and a command line interface. One way to do this is
to have two applications, and another way is DI (dependancy injection: pass
the "response" object around), however _another_ way is using dynamic
variables:

    
    
        D("output")(...);
    

This has the benefit of not requiring an extra argument to all of your
functions.

This happens more than you might think: Many people when confronted with all
their dynamic variables put them into some kind of "master object" (current
logged in user, output handler, database settings, etc), however dynamic
variables

When you do this, a lot of features become very easy to set:

• Capturing output (simply mock a new "output" var)

• Testing logic: dlet({db:test_settings},r)===dlet({db:live_settings},r)

• Impersonate users (if the right credentials are available)

And so on.

------
egometry
At IMVU we had feature toggles rolled into the A/B Experiment system. It
proved to be a vital part of Continuous Deployment.

Basically: Wrap every new feature in an Experiment Toggle. Deploy to
production (once ready for 'non sandbox' testing). Turn on for test users,
test. If good, start the rollout to experimental users. If after weeks the
numbers worked out, turn it on for everyone...

...but if at any point unexpectedly bad things happened, just flip the feature
toggle switch. Most of the time if "bad things" were happening in a relatively
new feature, this would stop the panic and let analysis of the root cause of
the problem happen without the pressure of everything being on fire.

~~~
prostoalex
Same at Facebook [https://www.quora.com/How-does-Facebooks-Gatekeeper-
service-...](https://www.quora.com/How-does-Facebooks-Gatekeeper-service-
work/answer/Jack-Lindamood) with rollout UI similar to what advertisers see on
ad purchasing UI (by country, college, demographics, etc.)

------
jammycakes
One point to be aware of with feature toggles: if you're using them as a
substitute for branching and merging in your VCS, you're deploying code into
production that you know for a fact to be buggy, insufficiently tested, and
incorrect. If your feature toggles themselves have bugs, or don't properly
isolate your new code, you could easily run into problems up to and including
data corruption. You may be switching your controllers on and off, but are you
also toggling your static assets such as HTML, CSS and JavaScript?

People are often scared of feature branches because of large, tricky merge
conflicts. However, large, tricky merge conflicts warn you (noisily) of
problems such as these before your code gets into production. Problems with
feature toggles, on the other hand, don't show up until after your code gets
into production, by which time it may well be too late.

------
cpitman
Overall feature toggles are an interesting idea. I'm not quite as adverse to
long lived branches as some seem to be, especially with CI systems that can
automatically detect merge conflicts when they first occur and run automated
tests on the merged code base.

However, this part seems strange to me:

> The team particularly appreciate that this will allow them to test their new
> algorithm without needing a separate testing environment.

Even with feature toggles, I would never want to test a new feature for the
first time in my production environment. Sure, the feature isn't on for
everyone. But it still can have a performance defect that brings the overall
system to its knees or a functionality defect that corrupts the data store for
everyone.

~~~
benjiweber
I find it amusing when CI tooling is used like this, to allow people to keep
changes isolated in separate branches with less pain from delaying the
integration.

It's more Continuous Isolation than Continuous Integration.

There are ways to test features for performance/accuracy in the production
environment with reduced risk. One approach I've used successfully is branch
by abstraction with verification.

[http://www.alwaysagileconsulting.com/articles/application-
pa...](http://www.alwaysagileconsulting.com/articles/application-pattern-
verify-branch-by-abstraction/)

Essentially you first extract a common interface for the component you are
replacing. Then you release both versions and send (all or a percentage of)
input events to both the old and the new implementation. You discard the
response from the new implementation and continue using the old codepath for
responding to users / performing calculations, but importantly - you compare
the old and new results and alert/log if they differ.

This allows you to gain confidence in a new implementation's behaviour in the
production environment and integrate your code with the rest of the system
with significantly reduced risk. When you're happy you can flip over to the
new implementation and delete the old.

There are also various other techniques for reducing the risk of datastore
corruption that you mention. I've written about some of them here
[http://benjiweber.co.uk/blog/2015/03/21/minimising-the-
risk-...](http://benjiweber.co.uk/blog/2015/03/21/minimising-the-risk-of-data-
damage/)

Most of these risks are actually smaller with more frequent and smaller
releases of functionality into production. Releasing more regularly also
encourages you to think about how to properly mitigate these risks.

------
eps
Oi vey.

Coming from the good old C I am frankly baffled how some triviality like this
one can be wrapped into so many fancy words and a full article. This is like
hearing that taking your shoes off when entering home is a good idea. Do you
know why? Let's start with an example. Picture this - it's raining outside,
water soaks the earth and some of it may get on the soles of your shoes... you
get the idea. It's a subject hardly worth a passing mention if any. "Feature
toggles". Bah. Makes you wonder what will happen when they discover function
pointers or, god forbid, the config files. Now that will likely be worthy of
at least a book.

------
quanticle
An interesting way to implement feature toggles is with your existing A/B test
system. Basically, create an "experiment" for each new feature. Set the
initial "treatment" size to something like 10%. Once the feature has been
validated in production, set the treatment size to 100%.

~~~
tamana
It's scary that you have to explain this. This is A/B testing 102, all new
features are experiments.

------
rbranson
A few months ago there was a great blog post by the Instagram team on their
particularly advanced flavor of this for their continuous integration. Highly
recommend:
[https://engineering.instagram.com/posts/496049610561948/flex...](https://engineering.instagram.com/posts/496049610561948/flexible-
feature-control-at-instagram/)

~~~
jessaustin
Whoa. That's an ugly URL! b^) I clicked through just to see if I could cut out
the random number, but no dice.

~~~
pkaeding
It seems the 'random' number is actually the important bit. You can remove the
`/flexible-feature-control-at-instagram/` part, which is only there for SEO
(presumably). StackOverflow URLs work the same way, in fact...

------
jbardnz
I really like the idea of feature toggles but i'm scared of quickly having a
lot of technical debt because of them.

I was looking at using something like
[https://launchdarkly.com](https://launchdarkly.com) but the pricing is very
expensive even at fairly small scale.

~~~
junto
We use feature toggled a fair amount on our SAAS application. It does create
technical debt which at some point has to be addressed.

One thing that confuses our junior developers is the difference between a
feature toggles and a client permission. Often they misuse a feature toggle as
a client permission, simply because only one client is using that feature
(currently).

~~~
justinucd
How do you typically address or mitigate your technical debt?

~~~
junto
Currently we are forced to ignore it until it becomes a performance problem or
a new feature touches the same code. This is simply because new features are
paid for by clients and they take priority over technical debt.

We only tend to refactor code (as a rule) when you are already working inside
that code section on something new (feature, improvement or bug).

Otherwise junior developers have a tendency to want to refactor all the
things!

If you wrote a new unit test, or altered an existing one to accommodate new
code, then knock yourself out and refactor away and knock yourself out!

------
evvvvr
This all thing reminds me why I love so much so called Enterprise Developers
and Martin Fowler with Thoughtworks and their articles (no offense, thanks to
Martin for the idea of refactoring, but such things are wrong). Seriously,
they take simple idea reducible to more general principles which is not a very
big deal and start build ad-hoc ideology and terminology around it: Hm-m, some
abstraction around feature flags config? I think we should call it "toggle
router". Then people start to create frameworks (not libraries) for this and
after all technical recruiters will end-up searching for people with
particular "toggle router experience".

------
bauc
Feature toggles are useful but should be used with great care, especially in
complex codebases. There are challenges to using them. If you aren't careful
or don't have very good test coverage you can break unknown sections of code.

The other pain is removing them, again if not done carefully or with very good
test coverage you can unknowingly break things.

------
celphys
I am pretty disapointed by the limited use cases pictured in the article. I
pinged the author without success.

Check at bottom of ff4.org page 10 others.

E.g : Deliver your product with toggles off on each node of your cluster and
when all are ready : toggle on - no more inconsistency during release - no
rollback as you simply toggle off if not working.....

------
tomcam
Ah, isn't this just... an option?

------
isbadawi
There was a nice talk about feature toggles at RubyConf 2015:
[https://www.youtube.com/watch?v=rBBLMmr9e-k](https://www.youtube.com/watch?v=rBBLMmr9e-k)

------
yosidahan
Configo.io can do that for your mobile app

------
yosidahan
Configo.io can do that for your mobile app!

------
omnibrain
It's interesting to see this here now, because ~10 hours ago, when it came up
in my feedly stream, I tried to submit it and only got the message that it's
already submitted. I searched here for it but could not find it, though.

~~~
omnibrain
Thanks for the downvotes for being curious how Hacker News works. :(

