Back in December I wrote the first of two posts noting the way in which some trends in our industry can rise and fall and then rise up again with all of the same promise, problems and fears. I noted the way in which the issues surround CRM and data mining seem to be having a second act in the big data movement. This is the second post and it's about the way in which the industry's drive to improve online questionnaires, knowingly or not, is drawing on some longstanding principles that until now have fallen out of favor.
One thing that most people in this industry seem to agree on is the sorry state of questionnaire design. We have redefined "good design" in terms of the things we can measure. So we count up the number of grids, page loads, words on the screen, etc. And at the end of surveys we ask the survivors how the experience they just plowed through compared to the vague benchmark of "other online surveys you have taken. " Back in the salad days of survey research we used the metaphor of conversation to describe a survey. The operative metaphor today seems to be closer to root canal.
But I see signs of hope. The first is what Jeffrey Henning has been blogging about under the heading of "Crowd-Shaped Surveys." The principle is simple: if you want questionnaires that work better for respondents then involve them in the design process. This principle is also a very old one. For decades researchers have used techniques such as focus groups, cognitive interviewing, think-aloud exercises, and pretests to fine tune survey questions so that they are easier for people to answer and yield the information that the researcher is looking for. These techniques are still widely practiced in academic and government circles but seldom used in MR. I guess we think that we just know how to write great questions. Jeffrey has been sharing a lot of ideas about how we can achieve some of the same goals via technology. I'm not crazy about all of what he suggests and look forward to seeing the validation studies, but it is one very promising path out of the rut we seem to be stuck in.
The second sign of hope is survey gamification, or at least some elements of how it is practiced by survey gamification guru Jon Puleston. (Jon also is an advocate of pretesting.) Now I admit that I'm no fan of Flash gadgets or many of the gaming features Jon has tested, but as I have written elsewhere one key element of Jon's approach is simply to write better questions by making them specific, creating a context, giving examples to clarify and trigger recall, etc. He reads the literature on questionnaire design and it shows in his work.
At the end of the day good questionnaire design is not about grids, page loads, interactivity, cute graphics or fancy answering devices. It's about using words well to ask questions that are clear and interesting to respondents and to provide a meaningful set of response options to capture the answers. That can be a tall order given some of the topics clients ask us to research, but there are ways to get there. It's encouraging to see MR remembering them.