Ask HN: Any good tips for writing/sending surveys?

[augustflanagan]

I'm writing a survey that I plan to send out to ~10,000 people in the next few days. They are all members of www.lenguajero.com and many of them are quite active on the site. While we get a lot of feedback via email we have never sent a survey to our members, and I'm a little unsure how to proceed.

The goal of the survey is to find out three things: 1. How useful is our website/how active are you on the site? 2. What features would you like to see added/improved. 3. Are there any features (existent or non-existent) that you would pay to use?

Does anyone have any suggestions for how to write a survey like this (i.e. how should the above questions be asked, and what other questions should be included in the survey), or can anyone recommend some good reading material about sending surveys?

Also, I'm not sure how relevant this is, but half our members are native English speakers and the other half are native Spanish speakers. Each member will receive the survey in his/her native language.

Thanks HN!

[Travis]

I work in epidemiology, specifically sending surveys to huge cohorts. This is what we've found through a lit review (sorry, don't have original sources but they're out there on PubMed).

- incentives are huge. But know if you're asking someone to complete it for a social benefit (e.g. our health surveys) or for your profit. If you're doing a "feel good about helping others" thing, don't offer things that can offend.

- make sure people know ahead of time what kind of questions and length of time to expect (properly set expectations)

- try to "group" similar questions. This helps people's brains load different topics, so their responses are easier (fewer context shifts)

- be aware that everything you do (colors, question wording, etc.) can significantly influence peoples' responses. You must be consistent with everyone who takes the survey (as much as you can -- you can't control the lighting in their rooms, as much as you'd like to).

- Make sure to be neutral in your language. Don't introduce bias by writing leading questions.

- try to use "standard instruments", i.e. other people's surveys that were validated and tested. Not only does this make your work easier, but you now have (the possibility of) a reference group to compare your results

- don't expect very many responses (this is from my experience on all projects). People are busy. Your survey is a small part of their day

- send the survey invite at least twice, spread by a couple of weeks (time permitting). Most people only complete their surveys if it's a convenient time to do so. Don't make them look back in their email -- send them a second invite (be careful beyond that, as you may annoy people).

- thank them very much

- have a detailed analysis plan before you create the survey. What are you trying to find out? Much like test driven development, knowing what you want to answer before you collect data is HUGE.

[dirktheman]

Make sure you offer an incentive for those who fill in the survey. Don't make the survey too long, but put a lot of thought into the questions and execution (sending it somewhere in the next few days to your entire user base might not be the best idea). Ask yourself questions first: what is is exactly you want to know, and to what extent are you willing to take action on the outcome.

[Travis → dirktheman]

This is so important I wanted to highlight it.

>Ask yourself questions first: what is is exactly you want to know, and to what extent are you willing to take action on the outcome.

[noahc]

1. Make it possible for them to take the survey in less than 60 seconds.

2. Only ask questions you actually care about (and have the power to do something about). If someone isn't active on the website or doesn't find it useful, they probably won't reply to the survey. Also, even if they did does knowing this allow you to make any meaningful changes at all?

3. With the features question, this is what I would do. Think of 3 - 8 features you think they want. Then put an other box. Have them vote on what the most important feature they want implemented is. If someone writes something in other give it a multiplier 3x or something like that. At the end of the day you'll have 1 or 2 ideas that lead the way. It'll be obvious what wins out, but don't make them think of what features they want.

4. I think you should ask, "Would you ever pay for features on our website" and just get a feel for if your typical website user expects to pay for stuff online or not. If the culture to pay for stuff isn't there, then it will be a hard go.

[kellyreid]

ask questions that give you answers to specific questions! dont ask how useful it is - ask what the most useful feature is. etc