

The $5 Guerrilla User Test - shalmanese
http://blog.bumblebeelabs.com/the-5-guerrilla-user-test/

======
AFerenci
I experienced the same effectiveness (95%-99%) of getting the "guy/gal reading
the newspaper" to volunteer for the study first hand. Although, this was more
focused in cafe environments.

Our team was trying to get beta users signed up for a location-based iphone
app in the Boston/Cambridge area and most people reading newspapers in coffee
shops or cafes are frequently willing participants. I even noticed that 7/10
times these people will engage with you in conversation regarding feedback/UX
improvements etc.

In parallel with Rodyancy's experience, some of the user feedback was
arbitrary (one guy started comparing our application to time/space continuum
theories...etc) but nonetheless, I would advocate approaching the newspaper
guy/gal as they are already on the knowledge quest and usually are more open-
minded.

Oh, and avoid the book/textbook readers. These people are either really
engaged in novel (imagine tapping on someone's shoulder during a movie) or are
stressed-out students.

------
rodyancy
I've done the sober equivalent to this (coffee shops). Not only did it point
out some usability issues, like, "oh, he thinks THAT'S a button," it also
proved to be a great way to generate ideas. Most of the input we received was
dead wrong, but we had to think about why it was wrong, and that got us
thinking in ways we hadn't thought before, which led to good ideas. I suspect
the drunken version would be more fun and just as useful, assuming you are
producing a consumer product. As for B2B, if your domain is specific, this
might not be the most valuable use of your time.

------
megamark16
What other types of guerrilla testing have people tried before? This sounds
like a fairly witty idea for gathering user interaction information, but I
wonder if it's actually functional.

~~~
jcl
I believe there's at least one website review service that is build on top of
Mechanical Turk, if that helps any.

~~~
megamark16
Thanks, that's good to know. I know someone once mentioned using Mechanical
Turk and that those sometimes ended up being their most loyal users, but I
cannot recall the context. Has anyone used these services and what results did
you see?

~~~
pw
I've tried Feedback Army and was disappointed w/ the results. Not that it
isn't a cool idea, but I think the give and take of a good usability test
requires that it be done in person. If you watch usability expert Steve Krug
doing a demo test ([http://network.businessofsoftware.org/video/steve-krug-on-
th...](http://network.businessofsoftware.org/video/steve-krug-on-the-least-
you)), you'll see what I mean.

~~~
raffi
I recently saw a blog post where someone wrote about the feedback they
received and they weren't happy. The same person then revised their questions
and tried a more task oriented approach and they were much happier.

[http://www.i-proving.ca/space/David+Jones/blog/2010-01-27_1?...](http://www.i-proving.ca/space/David+Jones/blog/2010-01-27_1?showComments=true#startComments)

Then again, Feedback Army is just a usability testing tool. The right one for
the job. I think it's best when a breadth of ideas are needed quickly or you
need some ideas about where to start improving your site.

You should use other methods to do testing that requires a lot of depth from
one or two people.

Here are a few more posts where folks have written up their experiences:

[http://visualwebsiteoptimizer.com/split-testing-blog/how-
an-...](http://visualwebsiteoptimizer.com/split-testing-blog/how-an-
investment-of-10-can-give-you-website-improvement-ideas-worth-thousands-of-
dollars/)

[http://www.mattmazur.com/2010/02/reflections-on-precedens-
ha...](http://www.mattmazur.com/2010/02/reflections-on-precedens-hackernews-
launch/)

(p.s. I run Feedback Army :))

------
coriander
So you want programmers to talk to people in real life. Interesting.

------
run4yourlives
This is popular in some design circles, particularly with evaluating reactions
to 2/3 designs prior to launch.

I'm not sure you'd get exactly the same quality of results with a user test,
and demographics is key - after all, the blue-collar workers at the pub aren't
the people to ask about your b2b conversion tool - but it's certainly the type
of thing that we should be doing.

~~~
shpxnvz
I'd imagine that for just about any demographic you care to target, there's a
bar you can find that caters to them.

Heck, I've seen bars that have clientele heavily biased to a specific
industry, and in some extreme cases, even a single employer.

------
c_allison
Not only a good way to test cheaply, but a good way to network! Win-win.

------
dkimball
I'm kind of disappointed that Steve Krug didn't think up something like this
for _Rocket Surgery Made Easy_
(<http://www.sensible.com/rocketsurgery/index.html>); this would be a very
convenient source of manpower for his model of "hallway usability testing,"
and would allow even larger numbers of even more easily set-up tests than the
model he describes.

It's good to see how usability testing is becoming de-mythologized these days;
all of us should keep in mind, for current and future program development,
that you really don't need a very sophisticated apparatus.

------
Groxx
_Start off approaching a group of the same sex as you so that the encounter
isn’t sexualized. Next, move on to a mixed sex group and then finish the night
with an opposite gender interaction so you get a nice demographic spread._

I wonder if that last bit is intentionally a double-entendre. Especially since
the proposed environment involves a bar.

It's an interesting technique, and I'll definitely keep the strategy in mind
for future endeavors.

~~~
shalmanese
The double entendre was completely unintentional. Testing against women is
important but one of the big hangups a lot of people have is the sexual
component. This is one technique I've discovered to help ease out of that
mindset.

As a strategy for picking up women, this is somewhere down the bottom in terms
of effectiveness. As it turns out, asking someone to participate in a user
study is about the least sexiest thing imaginable.

------
jdrock
One additional note that should be added: results will only be useful if your
desired customer base is represented well by people at a bar.

~~~
Spikefu
FTA: "Bars are wonderful at segmenting by demographic. Match the bar you’re
going to with the user population you want to target. Different bars will
produce slightly different results but the variation is not huge."

~~~
jdrock
Yeah, I don't know a bar frequented by people interested in the Semantic Web
or web-crawling.

~~~
ynniv
Doesn't sound like a consumer application. Given that your profile mentions
80legs, I think you're in the B2B space.

~~~
jdrock
Well, that was sort of the point of my original comment. The test will work if
bar-people represent your target customer base. This won't be the case for B2B
products and may not be the case for some Consumer products.

------
kolya3
Bring a laptop that you don't care about. Drunk people spill.

------
alexyim
Great. Now I have this to try in addition to the Parents Test

------
CoryMathews
This is awesome. I am going to start doing this all the time. Its extremely
good also that my target market is college students :) even better.

------
olegk
I don't see myself doing this.

------
ThomPete
I call BS on this. It has no usage what so ever.

You can't test for success you can only monitor it.

~~~
forcer
Why no usage what so ever? Any process of getting user's feedback is better
than doing nothing. We are so skewed of using our own apps that we think all
is perfect. Nothing ever is. I think its a great idea how to get more usage
data. If I try it I will let you know how it went. I am thinking of using some
screenrecording software like SilverBackApp to archive and analyze results
later when not drunk :)

Janusz Founder of <http://www.UsabilityTest.com> (free usability testing for
web apps)

~~~
ThomPete
Because success is not measured by the usability of your product.

There are countless examples of way too complex products that are still
successful and almost any successful company you can think of DIDN'T do
usability testing.

Companies like Amazon do A/B testing, Apple don't do usability testing at all.

Usability starts to matter later on, when you can measure on actual customer
behavior rather than user opinion.

Down vote me all you want. I challenge anyone to prove that usability tests
are actually making more successful products.

~~~
ThomPete
shalmanese

Amazon do A/B tests.

Apple doing usability tests? Not according to those I know there. Don't you
mean testing?

~~~
shalmanese
Amazon do both A/B testing & usability testing.

Apple's own HIG advocates for usability testing:
[http://developer.apple.com/Mac/library/documentation/UserExp...](http://developer.apple.com/Mac/library/documentation/UserExperience/Conceptual/AppleHIGuidelines/XHIGDesignProcess/XHIGDesignProcess.html)

Do you have a cite that either does not do usability testing?

~~~
ThomPete
Apple advocates it but don't do it themselves. They follow the genius design
method.

[http://www.businessweek.com/magazine/content/05_12/b3925608....](http://www.businessweek.com/magazine/content/05_12/b3925608.htm)

This article doesn't cite it but hint's at it.

Which is obvious for a couple of reasons. Apple is paranoid that anything
regarding what they work on get's out. and Apple use the Genius design method.
Apple do new things not me to products.

Other than that you must take my word for it and those I know at Apple. But
again if you can find anything that proves that Apple do usability testing
then by all means please share.

With regards to Amazon. I know that what matters at amazon is A/B testing.

[http://glinden.blogspot.com/2007/06/ab-testing-at-amazon-
and...](http://glinden.blogspot.com/2007/06/ab-testing-at-amazon-and-
microsoft.html)

And looking at their checkout process it doesn't look like anything that have
been anywhere near a usability test.

But you are missing the point here.

The companies didn't become successful because of usability testing that is
the crux of the matter and the claim behind the 5USD that is wrong. To the
extent it testing your product matters you can ask ANYONE. If it is that bad
then why spend 5USD on a beer.

But most likely it wont be totally unusable and then the actual evidence of
usability test having any positive influence on your likely success.

