Interesting, but not a very informative article. I'd be curious to hear more about the technologies behind the framework.
Is Airlock entirely server driven, or are there aspects of it (say, data storage or selecting which variation of a test to display) in the app code itself? Would it make sense to create a framework that lives entirely in Cocoa and only syncs with a server? Why or why not?
I helped build the server-side component of this framework during my first internship at Facebook. The framework is server driven, in that experiments are defined in another tool and then loaded on the client framework. Essentially, an experiment is parameterized by name and a user's ID. The server code then determines which experimental condition the user should be placed into, and it returns the values of the parameters corresponding to that condition.
The framework was initially designed for the web, in an effort to make a simple framework for running experiments with parameterized text content. Copy needs to be translated to launch on Facebook, as many people use the site in non-English languages. The framework integrates with several other internal tools to make running experiments with text a lot easier - mainly, the translation services and Gatekeeper for targeting specific countries, languages, etc..[0] The framework is also dead-simple for running experiments with other scalar parameters - as an example, testing the number of results returned from a search.
I'm sure you could write a framework that exists solely client-side, but then you also have to re-write all of the experiment logic for each platform, rather than the 'wrapper' required to sync server-side and handle edge-cases like delayed exposure, etc. The wrapper is already non-trivial, so it's also nice that you can ensure the core framework is well tested without having to worry about testing in different environments. As I said before, integrating with other services is also really great, and for Facebook, most services are in PHP or exposed through Thrift, so accessing them on the server is very straightforward.
Another upside to being server driven is that updating an experiment can happen on the fly, so the moment you determine your experiment's best variant, you can disable the other ones without waiting for Apple to review the app again, etc. I'm sure you could implement this with a client framework, but it seems less elegant if you are relying on the server for key information about the status of the experiment, but still trying to retain all the experiment logic on the client.
>> "Would it make sense to create a framework that lives entirely in Cocoa and only syncs with a server? "
I haven't finished the full article yet but as the Facebook app is available on multiple platforms I'd say it wouldn't make sense to develop the testing framework in native code entirely for each platform. Look how long it took for Facebook to give up on HTML and switch their app dev to native. If possible they would probably do the testing framework non-native.
Apple makes developers wait to release new versions (approval process) while Android does not. As a result, iOS is a better choice for A/B testing since it has longer cycles of deployment.
A/B testing has nothing to do with the application review process. It's for either A – bringing a new service online that cannot handle the load right away (server tuning), or B – testing the UX and fine tuning it before releasing it to entire user base (client tuning).
The issue is that any service at Facebook written in PHP inherently contains a lot of artifacts that are only relevant to Facebook's tech-stack. Services written in C or another language have a greater chance of eventually being open-sourced; and though I didn't mention it in my other comment, the server-side component of the framework is written in PHP.
Is Airlock entirely server driven, or are there aspects of it (say, data storage or selecting which variation of a test to display) in the app code itself? Would it make sense to create a framework that lives entirely in Cocoa and only syncs with a server? Why or why not?