For what it's worth, using Clutch was one of the reasons we ended up making Apptimize. We ran into several bugs including one that was crashing the app we were working on and issues with its probability distribution. And the reporting was very simplistic.
Are you using Clutch now? If so, it'd be awesome if you could give us a try.
Additionally, incumbent mobile A/B testing service http://www.leanplum.com -- I've used them for a while now and it's a pretty robust system with good support. Worth checking out IMO.
This looks interesting - although having never really considered using A/B testing on native apps before I did spend a few minutes looking around the site for examples of things to track. Maybe you could show an example project on the site showing what was changed and how that affected sales, etc.?
Part of apptimize team here. We want to do that! We will add more to our site; currently it looks deceptively slick bc our designer is awesome but it's really a first iteration.
Right now if you sign in, you can see an example experiment. How does it look to you?
We haven't announced our pricing yet because we're constantly adding new features our users request. Email me at nancy at apptimize dot com and we'll give you a special deal we've been giving to early adopters.
The lack of information on pricing is an instant back-button-trigger for me. I have no idea if you (eventually) plan to charge $5 per month or $200 per month. If it's in the lower end, then it can be worth evaluating. If it's a few or several hundreds per month, then I have to pass until I'm earning enough to afford it.
Do people actually do A/B testing on iOS apps? I feel like the two-week development cycle really hinders your ability to conduct valuable A/B tests with your users in time to actually make any change.
Some of the more forward thinking folks (Socialcam and HauteLook for example) have rolled in house versions of these, but Apptimize is one of the first end to end solutions I've seen. But A/B testing in mobile is definitely something a lot of folks are thinking about.
We faced the same problem, and built an A/B Testing Framework at a Hackathon this month that allows you to just drop in a line of code and run all the experiments through a web interface without app updates.
The framework is not quite ready to ship yet, but we built a fully working proof of concept, and are looking to complete it and put it out there.
You can read more and watch a video demo at the website: http://fliptest.io
Basically, it removes the 2-week development cycle, and you can essentially make changes in real time. I've seen it demo'ed, and it's much more impressive than the website makes it out to be.
You're right the app store is a barrier but people still do it. That's one of the advantages of our product, that you can start and stop experiments so you can put in as many variations as you want. Currently people just suffer through the really slow time frame.
We build in DB-driven tests into our app so we can make changes at will once we find a winner. Then we just clean up the code to remove the test in the next iteration.
If you register (takes 2 seconds) there's an example project where you can see the integration steps with the exact code you need to paste in. You're right though, we should definitely add examples to the main site.
I'd recommend not forcing people to register to see generic information like this - I know HN is biased towards "Connect with Facebook to login", I would assume most are also biased towards registering to see content - I know I am.
Part of apptimize team here! We let you track whatever goals you want, so people measure stuff like "user returned within a week" and "user used app." What goals do you want to track?
One of the biggest reasons is probably the reason behind using any software as a service rather than doing it yourself: you can concentrate on what you do best and not spend time managing things yourself. A couple of our users switched from using open source alternatives for that reason. We've put a lot of time into our infrastructure and decoupling things so we're very resilient against outages so you don't have to. We've also put a lot of time into making our library more robust and well tested than any alternatives we've seen to date.
I'm guessing because it's better than what they could integrate and build up with an open-source alternative in a short amount of time and thus more cost effective.
I've got some karma points to burn, so here comes something.
A list of your clients would be nice to see. So that I'd know to never install any of their apps.
I hate apps that phone home. I hate apps that phone to some random analytics companies even more. I don't like companies that enable, entice and profit from this sort of behavior, a behavior that every single user would opt out from if they were asked upfront. So, unfortunately, the only thing I can wish you at this point is a speedy demise so that you'd be free to focus on projects that are less ethically questionable.
How does the anonymous gathering of statistics harm you? As an app developer, this data allows me to measurably improve my apps.
I have several free (no ads either) apps in the Android market, and even in the most trivial of apps I will include Google Analytics to receive crash reports, see what screens are most popular (so I can improve my apps where it matters most), where the app is being used (for some reason one of my apps was really popular in France, I then added a French translation and the adoption really took off), what devices are most/least popular etc. If I add a new feature, I want to know if its being used or if I wasted my time, if I need to improve the discoverability of the feature etc.
Last but not least, the usage data is really motivating to me as a developer of free apps. If I release an app, and I see it being used daily, that makes me happy and motives me to keep updating it.
As far as I know, the GA data is anonymous and only shown to the developer in aggregates.
I agree with you 100%. What it ultimately boils down to is the need for feedback.
Publishing an app is a conversation with your users instead of throwing a message in a bottle into the ocean. You need all kinds of data and metrics not only to see how well your business is doing, but also how you can help users have a better experience with your app. For better or worse, until telepathic feedback is invented, phone-home is all we have.
> As an app developer, this data allows me to measurably improve my apps.
I don't give a flying f#ck about that, sorry. This is my device and I don't like my device connecting god knows where at a whim.
As I said, if you were to tell your app users upfront that you will be tracking their usage, anonymous or not, and gave them an option to opt out, very few would not opt out. This is the exact same inherent need for privacy that is fueling current NSA debacle. It may be irrational, but it pretty damn real. People don't like to be tracked or watched.
I think that frankly you are roundly wrong. Most users just don't care that they are anonymously being tracked or monitored. Most users what good, ease to use, reliable apps that do something interesting. If it phones home to google most people just don't care. The NSA scandal was mostly met with a loud "meh?" from most people. The media is pushing it as a story but really only techies give a shit about it. I'm not saying it's right but I'm pretty sure that's the general feeling about it with the general public. Anon analytics make better apps. It's a good thing for users all the way around. Better apps == a better user experience, that doesn't happen without better data.
How can that be debatable? If you see 45% of crashes happen on the photo-upload screen, isn't that some kind of clue where to look? I really don't think anyone can argue that analytics are not helpful in improving app quality. Please, if anyone can even play devil's advocate on that... please do. But you better make a throwaway account 'cuz I suspect you'll be buried into oblivion and ghost-banned.
You may want to consider a flip phone but even that has a GPS for 911... A/B testing is the least of your problems. If you are that concerned about privacy get off the cell network, root your phone with AOSP or your own OS and most importantly do not install any apps that you have not built. At that point consider A/B testing as it could greatly improve your user experience ;-).
Joke aside I understand your concern but this is really not the right place to start.
If he is using your app, it implies that he somewhat trusts you. I think it's Google that he doesn't trust. Using an app shouldn't require trusting a third party with personal behavior data.
The data as sent to Google (or any other analytics system) contains more than enough personally identifiable meta-data as to serve as a hugely useful database for the mining information on specific individuals.
Even just an IP address and device identifier is very useful metadata for assembling sweeping user profiles.
This creates a centralized data trove of hugely valuable data (why else do you think services collect it for free?) that is useful to both commercial and government interests.
This is why personal privacy matters, and why silently 'phoning home' without first getting explicit user permission is wrong.
Sending a request to a server is explicitly and inescapably part of requesting a web page; users explicitly grant permission for that request to be made obvious to you simply by making the request in the first place.
On the other hand, there's still an open question of what you do with that request data. Sending it to Google without permission is wrong, as it contains a slew of personally identifiable data that allows Google to build a massive profile of a specific user's behavior both across the internet, and in the real world.
