

GoogleBot as QA tester - johnbb
http://statsheet.com/blog/meet-statsheets-lead-qa-tester-googlebot/

======
fizx
I always "wget -r --header 'Cookie: login-credentials-if-neccessary' | grep
500" my staging site before sending it to production.

I also use hoptoadapp.com for error reporting.

~~~
fizx
Edit: For clarification, this isn't the only thing I do, but it's a helpful
sanity check.

------
nathanb
I agree that this method might work for statsheet.com, at least in its current
incarnation, but it's a bit of a special case. Just because you can rely on
spot-checking and only reacting to critical errors when you're strapped for
resources doesn't mean it's a good model.

It's like writing a post talking about how you've eaten beans and rice for the
past four months. Maybe you needed to do it that way to keep your company
afloat, but that doesn't mean that the post constitutes _advice_.

When bootstrapping a startup a lot of exceptional circumstances occur. Not
having resources to test a massive web of pages and data is one of them. The
last paragraph of the post makes this clear.

~~~
RobbieStats
Why doesn't it constitute advice? There are lots of entrepreneurs that are
strapped for resources. I don't find this as an "exceptional circumstance". My
point was simply that it is ok if you don't have 100% test coverage or any
automated test coverage. I think that is a waste of resources in a very small
startup in many cases.

------
ggrot
Checking for 500's seems like a pretty minimal test suite. There are plenty of
other errors that could occur.

~~~
RobbieStats
Sure, but the point is that when resources are very low you may need to get by
with the minimal amount of testing required.

The options are either to spend a bunch of time trying to squash every
potential bug, or know that some bugs will get through but at least users will
benefit from having a bunch of new features to play with.

------
WALoeIII
This only ensure the pages don't explode, you still have no idea that you are
showing NASCAR stats on your NBA homepage...

Crawl and Fuzz locally:

<http://github.com/relevance/tarantula>

~~~
RobbieStats
Actually I do because I'm the one that developed those pages. Why would I put
NASCAR stats on a NBA page? ;-)

Also, as a last resort users will tell me if something is off.

~~~
PhilChristensen
I think you should reconsider those statements.

Just because you wrote it doesn't mean it's doing what you think it is.

And users really can not be counted on to report errors. Even if they do, the
average bug report reads something like "So I put in your address into the
google, but when i got there nothing came up! I tried over and over, but every
time I clicked on the google, nothing happened! I asked my brother's boss'
kid, who's a real whiz with computers, and he told me I should 'eat my
cookies' or something, but I'm lactose intolerant!!"

~~~
RobbieStats
You are missing the point. At a small startup with limited resources you have
to make tradeoffs. The tradeoff I've settled on is more functionality in place
of 100% quality (if there is such a thing). I think people early on waste a
lot of time trying to nail every single bug when some bugs just don't matter
in the grand scheme.

Also, I'm not a junior programmer that needs someone looking over my shoulder
to make sure I've implemented a spec correctly. Saying it doesn't do what I
think it is doing is like saying that I may not exist even though I think I
exist.

~~~
btilly
I have seen _many_ senior programmers who think their code is doing X when it
is really doing Y. I've seen a very small number of programmers whose code is
always doing what they think it is doing. Every member of the latter group of
programmers spends considerable energy double-checking themselves.

Therefore if you're taking short-cuts, the possibility that your code is doing
something other than what you think it is is realistic, not ludicrous. Now it
may be _appropriate_ for you to take those shortcuts. But you shouldn't take
them without being honest with yourself about the risk.

~~~
RobbieStats
But what do you consider a short-cut? How much testing is enough? Doesn't the
situation dictate what is enough and what is really a short-cut? What needs to
be tested at a startup IS NOT the same as what should be tested inside a large
corporation.

And what "risk" are you referring to? As I've stated, I decided to side with
"more features" than "100% test coverage". Sure there is "risk" that there are
bugs, but a) there are bugs regardless of how much testing you do and b) I've
decided I'd lose users because I don't have features that stand out above the
competition rather than because a minority of those features have minor bugs.
So to me, the bigger risk is not developing more features.

