17/12/2012 Back to articles

The value of simple online surveys - and their dangers

Conducting primary research used to take weeks or months to plan and conduct. Now online tools such as SurveyMonkey and QuestionPro offer the ability to set-up simple questionnaires in a few minutes, and more sophisticated surveys in just a few hours. The free packages offer good quality, first level reporting and the data collected can be downloaded in full for the user’s own analysis.

This has revolutionised the data collection process and allows organisations of all sizes to conduct research quickly and at very low cost However, as with all research, there are potential dangers for the unwary. Common problems include samples being too small or skewed, asking the wrong questions in the wrong way, inadequate analysis and interpretation and, overall, over-reliance on, or downright misuse of, survey output. These can be avoided by following some simple guidelines.
Avoid the Barnsley screwdriver* trap
Use the right tools for the right job. Simple online surveys are great for getting quick feedback on specific, clear questions. If you need to test more complex issues where responses may be more nuanced an online survey is still valuable, but will need more time, care and know-how in its construction. Experienced professional help can help avoid time-consuming and costly errors.

Take into account who you are asking to respond
For example, sending a survey only to your existing customer base will not tell you the opinions of those who do not know you, or who choose not to buy from you. You do not need a massive sample, statistically representative of the whole universe, for every survey. But you do have to understand the restrictions on interpretation that a more limited sample brings. This is especially the case when presenting information to senior management teams who will, almost universally, extrapolate from the particular to the general if allowed to do so.

Pilot content, analysis and interpretation
Once you have put together your draft survey, run a test with a small number of trusted colleagues. Look at the reporting output from the tool you are using and/or conduct the analysis you intend on the downloaded data. Check if you can interpret what is coming out, and think about questions that arise. Go back to your participating colleagues and check if your understanding of their responses is correct and so understand if they understood the questions in the way you intended.

Sense check the results
The results of surveys may initially surprise management, for example we find pricing studies often suggest prices could be raised. With reflection it is usually possible to arrive at coherent hypotheses to explain results and still make rational recommendations, including further research if necessary. Failure to think through possible explanations for anomalies risks rejection by senior management of all conclusions drawn from the survey.
 Peter Wright
(* With apologies to our readers from Yorkshire – ‘Barnsley screwdriver’ is a Lancashire term for a lump hammer)