
Ask YC: Do you test your site? - riklomas
Have you used A/B testing or multi-variate testing on your site?<p>Did you find it useful if you did? What products did you use and did you find them easy to use? Was it effective and did you achieve the gains you expected?<p>If you haven't, why not?
======
Shadow84
Hi,

I was responsible for Web Analytics in a Web Agency in Switzerland.

In my opinion A/B and multi-variate testing only has a real use if you have
very high volume. If you have lower volume or are in a Beta or even a closed
Phase of your site development cycle I would suggest to use direct usability
testing where you watch people interact with your site and interview them
about what the think about your site. This way I think you would get much
better results for the same or less time and money.

Best regards

~~~
fallentimes
I don't think you necessarily need a high volume. I think you just need enough
that's statistically significant for whatever you're testing. And like another
commenter noted, google optimizer is free.

~~~
donal
Most definitely, otherwise you are really just eye-balling it and you don't
want to find out that your small sample-set wasn't representative of your
actual user-base.

This is important considering that during early stages you could have early
adopters and other more tech-savvy types using your site more heavily than
what you will end up with.

~~~
Shadow84
You don't necessarily need to give people a specific task.

For example, just ask them to discover the site, watch them in what they do,
watch what they spend their time with, discover which problems they have while
surfing your site. As always this can, of course, only be a part of your whole
testing effort. Watching people solving tasks is also important.

Additionally I have to say, that I am not against A/B or multi-variate
testing, in my experience it just is very time consuming (and therefore
expensive) and the effort to get results is bigger than with direct testing
IMHO. If you think you have the money or have exhausted the possibilities with
direct usability testing, A/B or multi-variate testing is a valid option.

------
DanHulton
Testing's _critical_. But there's multiple levels of testing, as far as
websites go. At the very most basic level, there's things like Google's A/B
optimizer. At the other end, there's things like Amazon's obidos - a tightly-
integrated testing/recommendation engine.

It's going to come to the point where if you AREN'T live testing on some
level, you're going to flounder and fall beneath competitors that are. And the
better your testing, the better you'll do.

(Disclaimer - I work for a Sitebrand, company that tries to offer a middle-
ground in this area. We do personalization, similar to what Amazon offers, and
A/B testing of various content. My opinions are very influenced by what I've
seen, which is that companies that sign on with us see significant growth
because they're putting in the work, the testing, and the personalization.)

------
cianchette
I've used A/B and multivariate testing at multiple companies. It can be
incredibly helpful in optimizing a product.

I would suggest however to focus on building a solid base product first and
then optimizing it.

For web the google web optimizer works ok and its free. There is a Boston
startup called Sitespect who I've used in the past as well, which has a more
robust platform. I can introduce you to them if you want.

------
craigbellot
I test everything, headlines, images, buttons, etc. If you want to know what
your users want, you can test your way there. Your users will tell you.
Especially useful for e-commerce and lead generation sites. If you can
increase conversions from 1% to 2%, you've just doubled your revenue. Google
website optimizer works well for me.

