

Airlock – Facebook's mobile A/B testing framework - dsr12
https://code.facebook.com/posts/520580318041111/airlock-facebook-s-mobile-a-b-testing-framework/

======
tannerc
Interesting, but not a very informative article. I'd be curious to hear more
about the technologies behind the framework.

Is Airlock entirely server driven, or are there aspects of it (say, data
storage or selecting which variation of a test to display) in the app code
itself? Would it make sense to create a framework that lives entirely in Cocoa
and only syncs with a server? Why or why not?

~~~
ampersandy
I helped build the server-side component of this framework during my first
internship at Facebook. The framework is server driven, in that experiments
are defined in another tool and then loaded on the client framework.
Essentially, an experiment is parameterized by name and a user's ID. The
server code then determines which experimental condition the user should be
placed into, and it returns the values of the parameters corresponding to that
condition.

The framework was initially designed for the web, in an effort to make a
simple framework for running experiments with parameterized text content. Copy
needs to be translated to launch on Facebook, as many people use the site in
non-English languages. The framework integrates with several other internal
tools to make running experiments with text a lot easier - mainly, the
translation services and Gatekeeper for targeting specific countries,
languages, etc..[0] The framework is also dead-simple for running experiments
with other scalar parameters - as an example, testing the number of results
returned from a search.

I'm sure you could write a framework that exists solely client-side, but then
you also have to re-write all of the experiment logic for each platform,
rather than the 'wrapper' required to sync server-side and handle edge-cases
like delayed exposure, etc. The wrapper is already non-trivial, so it's also
nice that you can ensure the core framework is well tested without having to
worry about testing in different environments. As I said before, integrating
with other services is also really great, and for Facebook, most services are
in PHP or exposed through Thrift, so accessing them on the server is very
straightforward.

Another upside to being server driven is that updating an experiment can
happen on the fly, so the moment you determine your experiment's best variant,
you can disable the other ones without waiting for Apple to review the app
again, etc. I'm sure you could implement this with a client framework, but it
seems less elegant if you are relying on the server for key information about
the status of the experiment, but still trying to retain all the experiment
logic on the client.

0: [https://www.facebook.com/notes/facebook-
engineering/building...](https://www.facebook.com/notes/facebook-
engineering/building-and-testing-at-facebook/10151004157328920)

~~~
tannerc
Thanks for sharing your insights and a little bit of what's going on behind
the scenes.

------
diziet
Would love to see Facebook release this, rather than just talk about it!

~~~
ampersandy
The issue is that any service at Facebook written in PHP inherently contains a
lot of artifacts that are only relevant to Facebook's tech-stack. Services
written in C or another language have a greater chance of eventually being
open-sourced; and though I didn't mention it in my other comment, the server-
side component of the framework is written in PHP.

------
badhairday
Oh, it's not open source. I'm disappointed.

------
kodeshpa
Checkout mobile A/B testing framework Vessel.io
([https://www.vessel.io](https://www.vessel.io)), Supports iOS,Android,
Windows, PhoneGap etc. Easy to use.

