
Adopting Feature Flag-Driven Releases - ZachNelsonSF
http://blog.launchdarkly.com/feature-flag-driven-releases/
======
alaithea
It's funny that this is just becoming a thing. I heard about flag-driven
releases from somebody in DevOps at Facebook last year, and how integral they
were to their release process there.

I ran my own SaaS as a self-taught, solo developer from about 2002-2008, and I
used to insert feature flags all the time to test stuff out. Later, when I
joined the corporate world and learned more industry standard best practices,
I assumed feature flags were just another bad practice I needed to leave
behind with my old PHP project. But come to think of it, it was very
convenient to:

\- test things on a small scale live,

\- preview features with the biz folks,

\- and it was almost always very smooth to remove the flag and make a feature
live, because it was already part of the program's overall control flow.

Funny the things you come upon out of ignorance, versus the drive to do things
some way because that's the way everybody does them, and the good/innovative
ideas that may fall through the cracks in the rush from the former to the
latter.

~~~
pjbster
I dabbled with feature flags once. We had... issues.. with our DBAs who, it
seemed, couldn't make any changes to a database without taking it offline for
a cold backup first. We floated the idea of feature flags as one way to
perform certain live deployments. The ops guys refused outright for reasons
which stank of protectionism at the time.

Time passed. I was tasked with rewriting a fairly complex piece of code and I
put the new code behind a feature flag. At the time, there was no facility in
place to support feature flags so I had to introduce one as part of the
deployment. It all went into a test environment but the deployment scripts
didn't set the config to enable it. This was deliberate because I wanted to
verify that we wouldn't change existing behaviour (i.e. break anything) by
deploying with the flag disabled. The testers, however, were looking for the
new functional behaviour and raised a defect.

\- when the project manager found out, he claimed that this was responsible
for a timescale slippage (the delivery into test was massively late,
naturally)

\- when my line manager found out, he told me I'd be fired if I tried a
"stunt" like that again

\- when the DBAs found out, they decreed that, henceforth, all database
updates which resulted in a "functional change" would be subject to the full
release process (i.e. cold backups).

So the experiment was a total failure. Or a total success. It depends on how
you look at it; one amusing moment was when the PM came out of a meeting and,
with some gravitas, suggested that we might have to rip out half the new code
in an effort to salvage the timescales. When I demonstrated that we could
achieve that in, oh, about 5 seconds by flipping the flag, the notion went
away. The guy was a contractor and I suspected he was just trying to create
more work to keep himself around.

To me, the biggest question with feature flags is managing the removal of dead
code. In this case, it was never removed because the task of deleting it would
trigger a full project release cycle and the company didn't see the point of
spending the money.

~~~
Ntrails
We used feature flags to update "live" code but using if conditions based on
hard coded test user/client ids. I wrote everything necessary to respect a
database flag instead but that never made it out of a test db whilst features
kept being added behind "id" based flags.

So now to remove said flags you just hunt through code for the special id
string and remove the "no" branches. Oh, and for Gods sake don't accidentally
make the wrong thing live.

Flawless.

------
ilikerashers
Feature flags should be used judiciously. They are better suited to frontend-
type changes, enabling a button or something visible.

Flags to enable a web service call or the like is clutter most developers can
live without.

I find architects perceive them as the perfect fix for release management. To
me, they result in too much config bloat with minimal value.

~~~
pkaeding
I agree that feature flags, like any tool, should only be used where it makes
sense. I disagree that they should only be used for visual changes though.
Often, backend changes can have performance impacts that are only visible at
scale, so being able to roll something out gradually can help there.

Also, a flag can be used as a permanent control to disable a part of the
system (like an external call) if that part of the system is having problems.
Imagine if you had a github activity feed integration, and you wanted to be
able disable that feature when a github outage caused that call to take a very
long time. Being able to easily disable that feature without needing to change
your code can be immensely useful.

nb- I work for LaunchDarkly.

~~~
yclept
How exactly does the api work? Is every call to check a flag making a HTTP
request to LD or does the ld client sync periodically?

~~~
pkaeding
No, every call to check a flag is not making any sort of network connection.
The flag ruleset (includes details like specific users that should get each
variation, or particular user attributes that should indicate a specific
variation, or a percentage rollout) is streamed to the SDK in a long-lived
background connection.

The SDK maintains these rulesets in memory, and when you need to evaluate a
flag for a given user, the ruleset is evaluated. This way, there is no I/O at
the time that a rule is evaluated, so there is no real performance impact. At
the same time, because of the way we stream the rulesets to the SDK, any rule
changes take effect immediately.

[https://launchdarkly.com/performance.html](https://launchdarkly.com/performance.html)
has some diagrams, etc, detailing this (sorry for the marketing fluff :)

~~~
yclept
Great, we will be looking at applying this in a new project of ours, and
replace our existing feature flag implementations in other systems if things
work out.

------
e98cuenc
When are those conditionals deleted? I assume after a few conditional features
it can be hard to make sure all the possible combinations work well.

What strategy do you use to decide when to permanently enable a feature and
simplify the codebase?

~~~
swalsh
We have an automated system that emails you after 30 days of a toggle being on
100%. After that, it emails you once a week. Soon you get bothered by it
enough you take it out. System works.

~~~
bengale
That's a great idea.

------
jovanni
I work at a company that uses feature flags heavily for releases. Even most
refactors are done behind a flag.

It works surprisingly well, especially when you have 20 developers all trying
to release different features at the same time, you then have a controlled way
to isolate enabling and monitoring each new feature one at a time.

It's generally expected that you clean up your flag conditions after things
have been tested. And that you write code so removing the flag flow is
trivial.

~~~
avita1
> Even most refactors are done behind a flag

How does this work without cloning vast swaths of code and saying:

    
    
        if (feature)
           loadModuleA();
        else:
           loadModuleB();

~~~
jovanni
On occasion you do that. But usually it's just a couple if statements in a
couple places with very little code duplicated.

Also, again, it's expected that after it's tested you delete the old path.
It's a short lived transitional situation.

Sometimes it's annoying. Overall it's safer when you're on a massive scale.

------
paulddraper
I really like feature-flag releases, but they are not without disadvantages.

The one that comes to mind is complexity. Ten features is a thousand possible
configurations of your application. Thirty features is one billion possible
feature sets for your app.

Ideally, features are independent of each other and work regardless of the
enabled/disabled state of the other features. But ideally code doesn't have
bugs.

All in all, I think feature flags are great. I just prefer to limit them to
the larger, more risky changes.

------
agentgt
We do this and we measure the use of the feature and health with a metrics
library such as codahale.

I highly recommend using an easy to use metrics library to test feature flags.

------
lifeisstillgood
I think feature flags are a smell if used for anything other than controlling
which users see which output.

If Inhave a feature that is, say a way to calculate a customer quote, turning
on that feature without managing the deprecation is poor. Most "features" can
be seen as services provided to other areas of the code base, micro service
style or not.

It's a nice idea but I worry

------
heathkit
It's weird to see Google best practices continue to leak and become the
standard.

