I think it's great he's quickly turned around a site that lowers the barrier to a specific problem. I'd just worry about the demographic makeup of the respondents.
While Turkers aren't all geeks getting good sensible feedback is a skill in itself. I would worry that most Turkers a) aren't you customers b) aren't demographically vetted.
That said getting independent opinions is surely great food for thought and that's a powerful tool.
I was worried about this too. I honestly had no idea what to expect when I first built the software. Then I ran it against my own sites and I started seeing very thoughtful feedback. This is when I felt I had something.
I also posed questions directly to the Turk community about what I could do to make the process better and get results quicker. They spoke, I did what I could, and I'm really proud of the results my customers are getting. I'm not just saying that either.
I'm a researcher by trade and have absolute sales call reluctance. I won't sell something if I don't feel it is worth the persons time to hear me or money to invest into the product.
While this little project may only go so far, I'm not afraid to put my name on it and say I'm giving people something of value for what I'm asking.
I'm a researcher by trade and have absolute sales call reluctance. I won't sell something if I don't feel it is worth the persons time to hear me or money to invest into the product.
Are you planning to introduce anything to help select the right Turkers to comment on a project? That would be worth having.
Heck it might even be worth having as a standalone "selecting applicants for usability testing" application.
While Turkers aren't all geeks getting good sensible feedback is a skill in itself. I would worry that most Turkers a) aren't you customers b) aren't demographically vetted.
That said getting independent opinions is surely great food for thought and that's a powerful tool.