
When Feature Flags Go Wrong (2016) - scaryclam
https://www.infoq.com/articles/feature-flags-gone-wrong
======
rdtsc
Feature flags can be dangerous because even an extra boolean flag (on/off)
means doubling all the possible configuration states. With just 10 flags you'd
have 1024 different combinations. Do you test all of them?

Sure, in practice not all features interact (so you can be sure on module A in
the backend is orthogonal to changing color of the UI buttons), but sometimes
you can't be sure and feature combinations and interactions never tested will
appear in production and break unexpectedly.

One common thing I noticed is that feature A and feature B are both tested by
toggling one on, doing a test, toggling it off. The test passes, everything
looks good, product is shipped. Customer then turns A and B feature on and
product goes up in flames.

I like the idea of auditing configuration changes (not just features others
config options as well). I had implemented that across a few products and it
saved the day during debugging. For example we noticed customer has been
toggling feature combinations we never tested, or they reported "product
stopped working" but looking at the they've been tweaking the hell out of it,
which made it stop working as we had configured it.

~~~
moonman272
Could you tell us more about how you did your auditing? Did you add more
logging?

~~~
rdtsc
Yap it was done using the logging facility with a separate rotation, log
forwarding rules, and such.

------
arwhatever
Once worked on a project where the feature flagging utility code was named
FeatureMasking.IsSuppressed("feature name"), which would lead one to wonder
whether it was the feature or the masking which was suppressed.

Worse, rather than cleanly blocking off full sections of code, it was common
to find feature flag checking embedded within multiline expressions like

if (a>b) && c.IsNotEmpty && FeatureMasking.IsSuppressed("whatever") && d <= e
...

