Customer Surveys are Flawed
Customer surveys have a list of flaws so long it’s a wonder anyone ever bothers with them. You can only realistically ask closed questions with a selection of fixed answers. The response rate to them is in single digit percentages. And if they take more than 10 minutes to fill in then 5% response rate is ambitious. Ratings of satisfaction in surveys have a very weak relationship with actual loyalty and buying behaviour. Stated future buying intentions in surveys are cheap talk that you’d be crazy to rely on in your plans. Ratings are massively positively biased because you’re surveying people who already prefer your product to everyone else’s.
And those are just the drawbacks if you do them well. You can add a whole bagful of garbage in, garbage out if you do a half assed job.
But They’re Well Worth Doing Anyway
Nevertheless, despite their manifold flaws, surveys give you insights that nothing else can give you. That’s just as long as you understand their limitations and know they’re just one tool in your investigative tool box.
A survey is one of the few ways to find out what customers are buying from other people, and how much they spend on those things, whether they realise you offer them, and if they’d consider buying them from you. Surveys can reveal what customers think of your different products, how much they expect their spends to change next year, whether they would consider you at all for planned new products, and whole host of other valuable insights. None of these are guarantees, and some are just strong hearsay, but they give enough insight to know what to rule out and where to place bets.
Surveys combine best with other tools, like in depth interviews, analysis of historical buying, and (if you can afford it) experiments. These give you a much more complete and solid foundation for your decisions than any one of these tools alone.
On top of this, the logistics and economics of customer surveys are great. They don’t cost much and only take a few days if you have your email database organised. They give you actual data to analyse quantitatively. And they’re easy to repeat regularly to see changes once you’ve done one.
Customer Survey Top Tips
If after all that you believe like us that surveys are a good idea, here are our top 10 tips about how to do one well:
Stay in the survey lane, and only use them for what they’re good for
Surveys give a broad but shallow picture of customers’ buying habits, product use, assessment of your products, intentions for the future, and positive or negative views about concrete new product ideas. They give you data but not understanding. If you want insight then use in depth interviews. For purchasing behaviour of your products analyse actual historical sales records. To test out a new product idea properly, run a trial.
Budget for single digit conversion rate
Depending on your customers and your relationship with them, you shouldn’t count on a response rate of more than ~5%. So if you’ve got 200 customers, you’ll get 10 responses, which is about as useful as it looks. There’s things you can do to increase it but you get the drift.
Don’t do a first time survey without speaking to some people first
Two or three exploratory conversations will change your ideas about what customers do and don’t know. They’ll make your handful of survey questions actually valuable.
Know there’s the trade off between number of questions, completion rate, and quality of answers
If you’ve filled in a long survey you know this for yourself. You give less and less thought to every question after about the 4th or 5th, and then give up if the end isn’t in sight after 7-8 minutes.
Don’t expect anything valuable from open questions
About 1 in 10 people will answer any open question, and less than half of those will be more than a word or two. If you want answers to open questions, use interviews.
Test it
The first version will have questions people don’t understand. It will also have questions that always get the same answer, categories than no-one uses, categories where everyone checks “other,” and poorly spaced ranges where customers always choose the top or bottom range. If you have a big enough population, an initial test survey will help you make the most of the 5 minutes you’ve got of your customers’ time.
Make it anonymous (unless you’re combining it with analysis of actual behaviour)
We see respondents being more forthright when they’re anonymous, reflecting their true position. You can always add categorisation questions (role, size of company, sector, etc.) if you need that information. And you can have an option for people to give their details if they want their answers to be known.
Make it as attractive as you can to open
No one is working through their inbox looking for surveys to complete. So if you lay out a formal email with “survey” in the title telling them all about the exercise, and how much you value them as a customer then either they’ll just delete it as spam or their firewall will do it for them. Make your survey request email a short request for a favour, with nothing like “survey” in the title. Also, look at your own inbox on a Monday morning and consider whether it’s a good idea to send out a survey then.
Take the answers in context
Surveys, even when they ask mostly factual questions, rely on judgement, memory and some speculation. So when 65% of your respondents say they expect their spends to grow by 10% or more next year, it just means that most people expect their spends to go up. It’s not a promise you can bake into a business plan.
Remember selection bias
The people responding to your survey are your customers. If 90% of them think your product is great, that’s no bad thing but all it means is that 90% of the people who chose you are happy with their decisions. That’s all.
As anyone knows who’s noticed election predictions, fans’ views on the ref’s performance after their team lost, or claims based on twitter polls, there are lies, damn lies, statistics and survey data. Taken alone they’re incomplete, possibly deceptive, and sometimes dangerous. Done well, and combined with other sources of intelligence, they give you insights that you just can’t get anywhere else.
by Steve Hacking and Jim Powell
