
The Most Radical A/B Test I've Ever Done - patio11
http://www.kalzumeus.com/2010/06/04/the-most-radical-ab-test-ive-ever-done/
======
px
As a teacher, I think this is a smart decision.

Desktop applications are increasingly difficult to manage on a school
computer. Many teachers return each September to a computer that has been
wiped clean.

At the same time, the internet is becoming more and more accessible in
schools. Teachers will be able to use the web application no matter which room
they are in or which computers they have access to.

Another challenge for non-tech savvy teachers is syncing work between the
school and home. I think focusing on the web app will mitigate some of the
anxiety that teachers experience in that area.

~~~
patio11
_Many teachers return each September to a computer that has been wiped clean._

I hear that. Frequently, in September, from several dozen teachers asking me
"How do I reinstall Bingo Card Creator on my computer which was just wiped
clean? Also, do I have to pay?"

 _I think focusing on the web app will mitigate some of the anxiety that
teachers experience in that area._

It totally solved that problem, where the feature I made in the downloadable
version to do the same thing has been a catastrophic flop. (Twenty people have
ever used it, and I still get emails asking to explain how to copy files onto
a floppy so that they can be read on the machine at school. And then emails
the next day saying that the computer doesn't know what a .bcf file is. And
then emails the next day saying that they installed Bingo Card Creator and
opened the file but it says it is the free trial which is odd because they
know they purchased the program and they never had this issue at home [i.e.
where they followed directions and put in their Registration Key].)

~~~
ErrantX
Teachers here in the UK have it really bad; a lot of the IT infrastructure is
farmed out to BT or other completely useless corporates. Either what happens
is they can't cope and so are dropped after a few months or they keep having
to replace all the poorly configured laptops.

My Mum has been through about 8 new laptops in the last 14 months...

(the best one is when the laptop leads all went missing [never found out who
did it] and their response was... ...to replace the laptops. _rolls eyes_ )

And lets not even start to talk about the restrictions imposed on these
laptops.... (I mean; I can understand it for the kids, but the teachers?)

------
travem
My first thoughts are that removing the downloadable version will work out
well by getting rid of the "Tyranny of Choice" and thus helping with
conversions.

Has anyone else tried this approach?

------
paraschopra
And I thought the post _had_ the results of this test :)

Anyway, interesting test and do keep us posted about the results.

------
markkanof
"How do I copy the files to my niece’s computer? By the way it is a Mac and I
use a Yahoo. Try using the web application."

This gem from the article really illustrates the added difficulty of
supporting a desktop application vs a web application. For this particular
request what OS she is running doesn't matter, but for many other requests it
could be very important to know. Given the technical sophistication of this
user, just figuring out what environment the application is running in could
be an arduous process.

------
michael_dorfman
Very interesting.

My initial thought (unencumbered by any actual knowledge) is that the folks
who download the client version (instead of sign up for the web version) don't
do so for any particular reason, so I don't think that removing the option
should actually hurt.

Sounds like a good fit for Bayesian probability, using the old numbers as
priors.

~~~
JoeAltmaier
Downloaded app can be used offline.

~~~
michael_dorfman
But the question is: is that really important to the majority of the users who
download it? Or are they acting blindly?

Now, it could be that there is a contingent of bingo-loving teachers who
absolutely prefer to have the software locally installed, but I'm suspecting
(without any good evidence, I grant) that this is not really a key decision-
factor for most of Patrick's customers who choose to download.

~~~
patio11
I believed for years that there was a group of customers who would pay for
downloadable versions when they wouldn't pay for web versions. In fact, I put
AdWords ads on many free web-based competitors, and those are my best
performing ads anywhere.

But I've come to believe, after listening to customers and seeing the behavior
in tests where I emphasize or deemphasize the options, that this doesn't say
that users want to use downloadable applications. It was just that users want
easy to use applications that don't suck. Next to the free web apps in 2006,
BCC downloadable was easy to use and didn't suck. Next to BCC downloadable,
BCC online is easy to use and doesn't suck.

------
fierarul
Isn't it odd to announce the A/B test when you've just started it ?

I mean, I don't think HN readers are in the bingo cards market, but still, it
feels like he might be influencing the results by blogging about this
beforehand.

~~~
patio11
If I woke up in the morning and had a shocking lack of worries, between two
software products and client work I could probably find something real to
worry about without having to manufacture worries out of the ether.

"Several dozen HN users will go to the website, see which variation they get,
clear their cookies until they get the right one, then pay me $30... _and then
cackle madly_ " is not a real worry.

~~~
fierarul
Again, I don't think blogging about this will influence your A/B test, because
your (future) customers and people reading your blog don't overlap very much
(or at all).

But, I do think publishing the A/B test does influence the outcome if they do
overlap.

You are talking about the $30 that you could get but miss the point: you might
get some extra money but in the process corrupt the validity of your A/B test
and make a bad decision afterwards, based on that result. Well, not that it
matters anyhow, as I'm pretty sure you have actually decided already to stop
doing the Java app.

I would like to have somebody that knows more about sociology and statistical
surveys explain if what I am saying makes sense or not.

~~~
jules
> You are talking about the $30 that you could get but miss the point

I think you missed his: it's pretty unlikely that anyone on HN will do that.

~~~
dhimes
Are you sure that only HN readers read the blog?

------
fdb
I love this article. And his article on why he's done making desktop software
is even better:

[http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-
web-a...](http://www.kalzumeus.com/2009/09/05/desktop-aps-versus-web-apps/)

------
tomerico
I've just searched google for:

Bingo cards Generator

Printable bingo cards

Create bingo cards

Bingo Creator

In all of them you are not one of the top 3 results. It might suggest that a
better approach would be to work on your SEO than to increase your ad money.

------
coolio73
Enough of this uninteresting Bingo Card Creating program. How do these posts
keep getting pushed up to the first page?

~~~
Grinnmarr
I just wish I had enough rep to down vote your comment :P His posts and
complete transparency about growing his business epitomize the type of
community contribution that brings me back to this site time and time again.

~~~
coolio73
printing bingo cards is a business any 12 year old with a computer could set
up. Couldn't you do this on Print Shop and an Apple IIe? Most tech stuff
posted on HN is the cutting edge of web apps. And if it wasn't, I wouldn't
read this site multiple times per day. Sure, his writing skills and analysis
are decent/good, but the content he writes about is laughable. He should spend
some of his blogging energy on developing a more interesting topic/business to
blog about.

~~~
Poiesis
I will feed the troll:

 _He should spend some of his blogging energy on developing a more interesting
topic/business to blog about._

Patio11 is not your dancing monkey.

 _Sure, his writing skills and analysis are decent/good, but the content he
writes about is laughable._

Laughing all the way to the bank.

 _Most tech stuff posted on HN is the cutting edge of web apps. And if it
wasn't, I wouldn't read this site multiple times per day._

Well, can't say we'd miss you if you stopped contributing, but I'd have to
take issue with the first part. At this moment, the front page has 30 entries
of course. I can't believe I'm doing this, but they are about Flash,
SproutRobot, Falcon 9, bcvi, HTML5, ideas, diamonds, deflate, BigPipe, Falcon9
(again), Titan, YourWorldOfText, fivethirtyeight.com, being wrong, Toxoplasma,
Soros, Readability, Google IO, Posterous, Steve Jobs, Wakemate, Amelie(), the
FTC, patio11's a/b test, chrome dev tools, twitter employees leaving, Apple
and HTML5, Bing cashback, a world cup calendar and John Gruber. I'll leave it
to you to determine if "most tech stuff on HN is the cutting edge of web
apps." I tend to think that most business stuff on HN is there to help educate
someone to run their business, generally technical in nature.

Finally, there's _printing bingo cards is a business any 12 year old with a
computer could set up._

Why haven't you, then?

