

Detecting bots with Javascript for better A/B testing results - tptacek
http://www.kalzumeus.com/2010/06/07/detecting-bots-in-javascrip/

======
edash
There was a Javascript split-test framework posted yesterday:
<http://news.ycombinator.com/item?id=1408637>. So I'm wondering (because I
don't know):

What are the advantages of using Rails vs Javascript to do a/b testing? And of
course, advantages of either of these "custom" approaches over Google Website
Optimizer?

~~~
patio11
With regards to GWO, see <http://www.bingocardcreator.com/abingo/compare> .
Google doesn't use GWO on any page of importance and neither should you.

On the general topic of doing A/B testing server side (with Rails or what have
you) versus doing it client side in Javascript:

1) Doing it in Javascript sacrifices one-line A/B tests, and that is a cost
I'm totally unwilling to put up with because it means I will be too lazy to
actually do A/B tests. (Take a look at the framework posted yesterday and what
they have to do two swap two divs. That is fairly representative. Note that it
gets harder, fast, if you start doing anything the least bit complicated.)

2) Javascript is not your friend for load times. A/Bingo and similarly
optimized server-side frameworks are, essentially, free in terms of user-
perceptible performance.

~~~
lukestevens
"Google doesn't use GWO on any page of importance and neither should you."

That's not true, Youtube used GWO to run a successful, massive multivariate
experiment on its US front page, which qualifies as a page of importance in my
book: [http://youtube-global.blogspot.com/2009/08/look-
inside-1024-...](http://youtube-global.blogspot.com/2009/08/look-
inside-1024-recipe-multivariate.html)

------
aarongough
Good post Patrick!

 _"You might have noticed that I assume humans can run Javascript. (My site
breaks early and often without it.)"_

I find my sites doing the same thing more and more these days. Do you find
that this has caused you any problems? I'm guessing that a large part of your
target audience for BCC is non-technical so they're probably a good test...

~~~
iaskwhy
You need to be fairly technical to disable javascript. That or there's an
administrator doing that for you (like in some company).

So, besides companies which restrict javascript (very rare?) there's only
those using noscript (very rare?).

~~~
patio11
There are also a variety of mobile clients, but people don't try printing
bingo cards from their cellphone that frequently so I generally don't worry
about them.

More broadly, I tend to think that anyone who goes into their browser settings
marked Here Be Dragons is a lot like someone who routes my domain to localhost
in their hosts file. Ah, yep, that will indeed cause the site to not work. You
want the site to work? OK then, I would suggest not doing that.

------
pierrefar
This is not detecting bots (as per the title), but detecting user agents that
execute Javascript and are assumed to be human.

Detecting bots with JS knowing they don't run JS would be quite challenging.

~~~
patio11
Assume user #1234 requests a page with an A/B test on it. We hand them the
page and a Javascript challenge, and mark them as a bot. If user #1234
successfully passes the Javascript challenge, which if it happens will almost
certainly happen within the next few hundred milliseconds, we revise our
opinion about their character. Anybody who fails to convince us remains
flagged as a bot.

~~~
mike-cardwell
I don't know if it is currently being done, but it wouldn't be too difficult
to write a bot which uses webkit as a backend and supports javascript.

For an example, <http://code.google.com/p/wkhtmltopdf/> is an application
which converts a webpage into a pdf and supports javascript.

------
qeorge
Reminds me of hashcash and other proof-of-work challenges.

Nice post Patrick!

