

Ask HN: Web testing is killing us - best practice? - yeti

Hi, we're a startup with a fast evolving web game and are releasing new feature updates every 2 weeks.<p>Right now it takes us a solid 2-3 days for 2 people to test through the site with all existing and new features... testing isn't hard, it just takes ages and reduces time we can do other stuff.<p>Question - what should we do?<p>Suck it up and keep doing it this way?
Outsource the testing?
Motivate some of our hardcore users to test for us?
Try to build some automated test bed?
======
lucumo
Do you really need to be that thorough in your testing? What are the costs
when a bug gets into production and needs fixing? What are the costs of
testing and fixing it before it goes into production? Balance it.

We're only doing developer ad hoc testing. We just click through a few things
when we develop a feature (something you're doing anyway, I assume) and then
put in production. We don't do exhaustive testing. If an edge case doesn't
work and it gets reported, we'll fix it then. This has the added advantage of
only working on problems customers really consider problems.

This works because a web application can be pushed online very quickly, so you
can make many iterations and fixing a bug isn't inherently harder when it's in
production or in development.

This is a huge YMMV-topic, where you should really think of both sides of the
equation. Something that doesn't happen too often in Internet-articles, I'm
afraid.

~~~
yeti
We started this way and was fine at the start, but problem is that we have
many 1000s of customers now, and any bug that affects service causes us lots
of hassles to clean up after.. (we had bad email problems and currency issues
last month which took us a few days to recover from)...hence the need to test
at least somewhat thoroughly..

------
mtrichardson
Definitely an automated test bed, ideally runnable by just one command and by
every developer, and then again by Buildbot when you check in code (we pushed
a lot of our Selenium tests onto just Buildbot since we didn't have a lot of
js and they took much longer than anything else to run).

If you don't need a lot of javascript testing, look at Twill -
<http://twill.idyll.org/> . Nice and simple (and very fast).

Otherwise, <http://www.getwindmill.com/> and <http://seleniumhq.org/> .

------
nostrademons
Pull as much code as you can into self-contained, testable libraries that
don't interact with the outside world except through defined APIs, then test
the hell out of them with xUnit.

For everything else, there's Selenium.

