Can we choose, or at least influence the demography (only women, only people below 35, etc.)?
Unfortunately, can't choose demography but we try to break out the responses in a bunch of different ways so you can visually filter which demographic you're interested in.
I'm not saying that choosing your own demographics, number of participants, more complex surveys might not be useful sometimes. But I love that I can get my head around this in 10 seconds.
a follow-up question - how much more compelling would this be if you could choose/influence the demographics and # of responses (potentially at a higher cost)?
When it comes to consume research, demographics are everything. Statistical significance of sub-samples can plummet quickly.
If I'm marketing a product specifically to, say, 30-44yo men, it would be valuable to restrict the sample exclusively to those men and I think i'd have a better chance of seeing statistical meaning in the sub-samples I'm looking for, eg, income, marital status, etc.
It seems to me now that I'd have to post it "at large" and the 30-44yo demo I'm looking for would itself be a sub-sample.
I am not a marketing professional, but as a small business owner its one of my many hats. I have used both Maritz and, more recently, Pinecone, and I'm probably wrong but I don't remember even being presented with an option to open the surveys to everybody.
And what I'm saying is.. I wouldn't even consider using your product without that ability. You have a fantastic 1.0, though, so I'm not knocking the product, just offering my $0.02.
And some demographics of your own: We've got a dozen of us on the payroll with 2009 sales looking to be a bit over $2MM.
And no joke, if I were you, I'd eat your dogfood and put up a survey to determine the importance of this feature!
I do want to add that I get that your site is supposed to be intro-level market research. But if you're going to limit to 50 replies, demographic targeting would be very important to me, for anything other than a "should we get thai or pizza for lunch" questions.
If you do let me collect 500-1000 replies, I'd probably have more statistically meaningful sub-samples, so as long as you give me a nice drill-down interface I'd find it useful, even if i couldn't target my question.
I pretty much just fall asleep when I start trying to deal with A/B testing, market research, ad effectiveness, etc. This is so simple that all I have to ask myself is, "Do I have a question I'd like to ask a bunch of people?" and "Do I have five dollars?" That's a really good place to be, because everybody in business has questions, and everybody has five dollars.
I will absolutely use this site. I do question where the people are coming from, which I guess is a demographic question...but I think any time you ask of the user to choose it should be later in the sales process. For statistics wonks, mention it in the FAQ, or something. But I wouldn't clutter up the initial offer with too much stuff like that. I wish I had a product that was this simple of a sell.
And, of course, the more involved things get, the more expensive it will have to be. I really like five dollar market research. I might not be so enthusiastic about $25 market research.
I can ask usability questions of my users, but their existing knowledge will effect what is "easy" and "intuitive" (intuitive generally means, "what I'm used to"). I can ask marketing questions of them, but I've already pulled them into my site, so my current message will be the one that gets reinforced. I can ask "compare these two products" questions, but I already know a preponderance of people on my site prefer my product to our competitors (one of which is Fantastico, by the way). I can ask any number of things, and we do frequently ask our customers opinions on things, but by virtue of them being on our site the results are known to be skewed in exactly the way we don't want them to be for a large number of questions.
Also, I don't want to clutter up my site with constant polls. When someone is on our site, we have very clear goals we want to achieve: Teach them about the benefits, help them choose the right products, show them how to use it, answer their questions. Asking them random questions isn't my idea of good design, when it comes to those goals.
Examples 2-100: A poll on my site or blog would work fine but... I don't want people on my site to know what I'm polling. I don't have a site. I don't know how to do polls on my site. My IT department says it will take 8 weeks. My web design guys say they don't have the module for this particular CMS - they can build one from scratch for $2k. I told the web design guys to go to hell & found an embeddable polling service, it'll take 4 weeks & they won't be responsible if it doesn't work. I worked it out with the web design guys but my boss says we need to use the companies standard colour scheme - the widget won't change its colour.
I completely agree that there needs to be some way to restrict who answers questions.
Another example would be if someone was making a Twitter application, they might want to restrict people who answer the question to only those that use twitter.
Or if you were doing something aimed at webmasters, you might want to restrict the responses to people employed as such.
As also mentioned in other comments, use your poll system to do your own poll on how desirable this could be.
I thought of using this as part of a sales process for consulting services. That would put you in some sort of competition (or symbiosis) with compete.com. For that, immediately obvious conclusions (however simple), low price & easy to use (time is an issue here) are all essential. precision, advanced options & such, not so much.
Also, I just emailed a friend about this and shared the PickFu URL for the results of my photo survey on facebook, which according to Derek Sivers is a great sign (http://sivers.org/purplecow).
Finally, note that you can see all the PickFu questions as they are distributed to Mechanical Turk (even the "private" ones, obviously) at
I hope your site can scale. I predict growth.
Thanks for the heads-up about the mturk job search - we're making tweaks so that those previews show a sample question from now on. Helps keeps the responses more random too.
Thanks for sharing it!
1 vote with $ = many votes without $
I know you're going to say something along the lines of "we want to ensure that users are unique, and demographically diverse, blah blah blah."
Yeah, I know, but still. When I see "Mac or PC", I instinctively click on the "Mac" option, and when nothing happens, I think "Well, THAT's clearly broken!"
I know you probably don't want to create just another hot-or-not site. But, since you have public commenting that's separate from the "official" responses, why not also have public voting that's segregated? Maybe if I give you my demographic info, you could let me answer questions.
The bonus is that you don't have to pay me ;) So, perhaps you could offer a few options: $5 for "valid" demographically-distributed statistically-significant mechanical turk answers, or you can get all the answers you want for free from the PickFu.com community (still organized by demographics.)
Similar concept but a bit more information from the Turks at a slightly higher cost.
The Amazon VP of Mechanical Turk is going to come by to answer questions, and I'm sure she would love to talk to you all.
I don't have much experience with mturk, but my one concern would be that respondents may simply click randomly rather than honestly answering the question; do you have any data on how honest and accurate your responses are?
For example, we used this feature to dogfood our own FAQ page: http://pickfu.com/4UQM7U
the FAQ example is a bad example: I can always answer "yes" and then supply as reason "every questions I had was answered". No need to read the FAQ page, of course.
I think if you want to insert any kind of validation system you would have to count the time it took for the turk to answer your question and make sure we are in that range for every hit.
I'm sorry you had a rough experience with that specific music-listening job - we've never had a more time-consuming question submitted. Most are in the order of "which site do you prefer?", like this current one: http://pickfu.com/AGSIBT. These get answered far faster by the MTurk workers.
What would be truly amazing if this service could work with non-mechanical turkers...
Not to rain on your impressive parade, but I'd consider the type of person that wants or needs to use MTurk to be a pretty skewed population too, just in a different way.
Not saying it's not valuable, it may well be, and certainly more honest than asking friends - but I would expect a fairly low-income or bored-teen demographic.
Actually, an interesting meta-survey might be to survey the MTurk population itself, see what they think, their incomes, political beliefs etc; might be an interesting use of this very service.
But enough nit-picking - congratulations on a great idea, swiftly executed.
a few people have done this already
You're underselling yourself IMHO. Position your service as a more premium offering.
I'd do $50 for 50 comments. At least.
- At $5, you could try it out & develop trust as appropriate. This isn't market research with a consultant who can guarantee anything. This is DIY & you decide for yourself what it's worth. There's room for both kinds.
- Neither 5 or $50 is going to provide high quality trustworthy anything. That's not the point of this. The point is something interesting & potentially useful at a price that means you don't need to think about it too much.
1) Free, limited number of responses (maybe 5/month?), ad supported.
2) $5, Unlimited number of responses, no ads.
3) Skinable version for companies, allows them to manage questions/responses asked by employees.
We found that 50 responses offers a good set of opinions without taking too long to finish. What are some other options you'd like to see?
- The demographic segmentation just doesn't work with 50 answers. Maybe package it (say $5, $20, $50) & either only offer segmentation to the higher ones or recommend against reading too much in to the lower ones.
- Host the images & otherwise make the image part easy. I'm guessing this is an upcoming feature. Since you clearly set out to make market research cheap & easy, this is part of that. You'd be surprised how many people don't know an image has a URL. Also make it easy to get online images from websites without knowing images have URLs.
When I go to an example page, it says something like, "Survey says: B! (18 of 33 answers)"
And then I never remember which was A and which was B, so I have to look.
Why not "Survey says: Gmail!" or "Survey says: Obama" instead? Especially for the text ones. For the image ones it kind of makes sense to do A/B since they don't necessarily ever tell you a name for each image (though you could change that), but for the ones that are just text, it would be way more user-friendly to put the name in.
Just wondering, how has RPX ( https://rpxnow.com/ ) Login outsourcing/integration worked for you guys? Would you recommend it highly?
Thanks for the privacy warning - we'll look into it.
your domain looks like :: PickFuck :: to a casual glance.
Similar to how the FCUK brand (intentionally) looks like FUCK.
I think your site is good, although it seems the demographics are exclusively American. Hopefully you could add some multinational demographic later.
Good luck and well done!
The world is a big place, and competitors mean the market exists.