Hacker News new | past | comments | ask | show | jobs | submit login
Show HN: Easy A/B testing for your mobile app (open source) (keepsafe-engineering.tumblr.com)
32 points by philippb on July 31, 2012 | hide | past | web | favorite | 6 comments

Not bad. I have built a superset for my own iOS apps:

  0. Collect app environment (os/hw version, jailbreak, etc). 
  1. Collect runtime trace from the app.
      Inside the app I can trace beginning and end of 
      a function, optionally with a timestamp,
      as well as any atomic value (string, int, etc)
      The data is censored to remove any personal or private 
      information, (as defined by the app itself).
  2. Collect crashes (PLCrashReporter).
  3. Record assertion failures (the app keeps running)
  4. Submit the above to my AWS server.
  5. Receive configuration from the server. 
     It's a set of variables, boolean ones can be used 
     for A/B testing or block new (buggy) functionality,
     strings are used to change API endpoints etc, 
     ints are use to change timeouts or limits inside the app.
  6. The server indexes log files and promotes important
     pieces to the database, and brings them up to my attention:
     - crashes
     - asserts
     - identity (UDID, IP address, cookies, accounts ids, etc)
        useful for tech support
     - uptime, start count

At one point I also had a facility to record some usage stats describing user data and behavior (e.g. number of records, time spent reading "about" pages, etc), but I had to rip it out during a redesign and I am yet to add it back again.

Nice of you to open source it. I haven't looked over your code, but some thoughts come to mind.

Security - Make sure this doesn't lead to a back door into people's devices. Or make sure someone doesn't utilize this to make your product undesirable.

User opt in - Give users the ability to opt in or not. Tracking information might be useful, but some people do not like it.

Rollback is an important feature. User control of this feature with notification is a bonus. I.E. notify the user an error occurred with the option to rollback to a release that was stable for them. Currently we provide a link to all past versions which they can download and install over the current version if there is an issue. Not the best solution, but it has worked for us.

Internet access for applications that do not need it. This is tricky, so providing another service or plugin might be the way to go.

Automation - Manually keeping track of problems and resetting or rollback features can be time consuming. An automated approach with user opt in would be great as well. Automated analytic info on top of it would be even better.


Thanks for the great input. I Agree on some points and disagree on some other.

Security is something you have to be aware. The easiest way is to just setup the server with https so that the communication is secure. This seems to be the recommended approach for client server communication that Google them self suggested during the Google I/O app security session.

We currently don't use Switchboard for anything where you can manipulate the behavior of the app. Remember, the code that gets executed is still in your control when you build and sign the binary.

We focus a lot on streamlining the development process. We're only two people cranking out code all day. For this reason we want to have one version out there that we're able to maintain. We test the app that it runs on as many devices as possible and crashes as little as possible. The problem with providing old versions is that users then also have support request about old versions. At this point we're getting about 400 emails a day. When a user has an old version, we send him an auto reply to update to the latest before we deal with the problem. Otherwise it's not getting manageable.

I think Internet access is by now default for close to every app. You can always justify it in the app description with Ads, even if you never show a single ad in the app. Another option is to track crashes. Here the same again as before, if a user does not like that the app needs internet access, he should just download a different one. The reason here is that you will not be able to improve your app ongoing if you can't measure what happens.

Linking the analytics together with tests or stage roll outs back to an automated configuration tool is super cool. But again, we're 2 people and we have a pretty good overview of what we're running and what to look at. But I agree that this becomes very powerful when your team size gets bigger and not every engineer is keeping track of all business metrics.

You are very welcome. Thanks for replying back.

With regards to security. By setting up a method to access your application via the network means you have opened a potential security hole on the device. A port is opened and listening which means different types of attacks can take place given this port being opened. HTTPS doesn't secure a potential buffer overflow attack on your open port. In addition your back-end server can be attacked as well with people gaining access to data you have collected. Mobile devices are not well known for setting up firewalls and these are reactive defenses at best.

Switchboard is a great idea and again it is great that you have opened sourced the code.

With regards to manipulating the code. I guess I assumed incorrectly, I thought you used Switch board to have the ability to change code paths within your applications? I.E. use new source that has changed, or old source which has been tested and is working.

Internet access has drastically changed the retention of my applications. Currently 700K downloads with a 50% retention. I would say I could bump this retention up if my apps didn't have internet access when they really do not need it. (Currently ad supported apps).

I completely understand the two people dilemma. I am bootstrapping with only myself doing everything. I still need to do a huge make over on my pathetic web site, and I haven't started on any of my real scalable/growth ideas yet.

And again, thanks for sharing this... Cheers...

Next step, make it smarter than blind A/B testing.


nice work :)

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact