

Ask HN: how would you A/B test iPhone apps? - alex_c

patio11's A/Bingo post got me thinking.  How would you go about doing A/B testing for iPhone apps?<p>The issue, obviously, is the app store submission/approval cycle which makes changes slow to reach users.  (even without that, you probably wouldn't want to make users update their apps every day).<p>If the app is mainly web-based, it's pretty obvious, but what about native apps?
======
bdmac97
I know very little about iPhone programming so forgive me if this is ludicrous
but just thinking out loud...

We had an app launch on launchly awhile back that was basically trying to be
the iPhone version of GetSatisfaction or UserVoice. You could plug in their
code into your app and it would handle feedback and bug reports for you.

Maybe there's a market for something similar for A/B testing on iPhones. If
it's a tricky problem to solve, even better cause then people will pay you for
it.

Initial thought w/out knowing anything about the security model is can you set
up config files or even remote code on a server somewhere and then access it
from your code on the iPhone using an API key? If you can then you could maybe
set up multiple config files on the server and have your code load different
versions randomly and report back statistics...

Of course that brings up an interesting point. What would you be A/B testing
in an iPhone app? The user has to have already downloaded/installed your app.
At that point there's really not any landing page or funnel to test like there
is on a website. Obviously you can still A/B test internals just to improve
the app which will affect ratings which can affect downloads (conversions).

Anyways, just thinking out loud here.

~~~
zzz
do you remember the name of the iPhone support app?

~~~
bdmac97
Tap4Help

------
frosty
If the screen which you want to A/B test is not the first screen, you can make
background calls to your server and based on result decide which screen to
show. Write to user settings when you want to permanently disable it.

Note:

i) Reasoning behind not having it as first screen as it really slows the
application launch.

ii) You should make sure that if webservice call takes too long, just pick one
default which is shown to user instead of waiting for result

iii) if you really want to do it for first screen, load and save the result in
user settings for next launch while the application is running

------
ryanpetrich
It's actually pretty simple to A/B test on App Store apps:

1\. Prepare two versions of each feature you want to test

2\. Generate random integer on first launch

3\. Save to NSUserDefaults

4\. Present A/B based on the value of a particular bit in the random integer

5\. Collect statistics and send back anonymously via NSURLConnection

Question is probably better suited to StackOverflow

------
DougBTX
Guess:

\- Do fake A/B testing for your AppStore product page, toggle once a week and
compare sales.

\- For the product itself, try usability testing face-to-face with potential
users, using ad-hoc distribution.

------
wmeredith
Design an iPhone app that's a CMS pulling in data streams (XML) and displaying
them to users. Set the first queried data stream to be the A/B switch and all
other data is pulled into the app dependent on the state of that switch. This
is all assuming you're talking about testing within the app itself after it
has been downloaded. (I'm an SEM/SEO UX guy, so this is out of my league, but
I think that makes sense.)

