The Survey Geek

Plus ça change – Part 2

Back in December I wrote the first of two posts noting the way in which some trends in our industry can rise and fall and then rise up again with all of the same promise, problems and fears. I noted the way in which the issues surround CRM and data mining seem to be having a second act in the big data movement. This is the second post and it's about the way in which the industry's drive to improve online questionnaires, knowingly or not, is drawing on some longstanding principles that until now have fallen out of favor.

One thing that most people in this industry seem to agree on is the sorry state of questionnaire design. We have redefined "good design" in terms of the things we can measure. So we count up the number of grids, page loads, words on the screen, etc. And at the end of surveys we ask the survivors how the experience they just plowed through compared to the vague benchmark of "other online surveys you have taken. " Back in the salad days of survey research we used the metaphor of conversation to describe a survey. The operative metaphor today seems to be closer to root canal.

But I see signs of hope. The first is what Jeffrey Henning has been blogging about under the heading of "Crowd-Shaped Surveys." The principle is simple: if you want questionnaires that work better for respondents then involve them in the design process. This principle is also a very old one. For decades researchers have used techniques such as focus groups, cognitive interviewing, think-aloud exercises, and pretests to fine tune survey questions so that they are easier for people to answer and yield the information that the researcher is looking for. These techniques are still widely practiced in academic and government circles but seldom used in MR. I guess we think that we just know how to write great questions. Jeffrey has been sharing a lot of ideas about how we can achieve some of the same goals via technology. I'm not crazy about all of what he suggests and look forward to seeing the validation studies, but it is one very promising path out of the rut we seem to be stuck in.

The second sign of hope is survey gamification, or at least some elements of how it is practiced by survey gamification guru Jon Puleston. (Jon also is an advocate of pretesting.) Now I admit that I'm no fan of Flash gadgets or many of the gaming features Jon has tested, but as I have written elsewhere one key element of Jon's approach is simply to write better questions by making them specific, creating a context, giving examples to clarify and trigger recall, etc. He reads the literature on questionnaire design and it shows in his work.

At the end of the day good questionnaire design is not about grids, page loads, interactivity, cute graphics or fancy answering devices. It's about using words well to ask questions that are clear and interesting to respondents and to provide a meaningful set of response options to capture the answers. That can be a tall order given some of the topics clients ask us to research, but there are ways to get there. It's encouraging to see MR remembering them.

Thinking or just plain stumped?

Last night I had dinner with an old friend who also is a world renowned and widely published expert on questionnaire design. We chatted some about what's happening in MR and I asked for his take on the industry's obsession with speeders, i.e., respondents we decide have answered questions too quickly. Or, more specifically, does a longer response time signal a more thoughtful and better answer? His first reaction was to suggest a U-shaped curve with the sweet spot in the middle. A really fast response might indicate no thought at all while a longer lag in responding may signal some problem in coming up with what the respondent thinks is an appropriate response. Maybe the question is confusing or the first answer the respondent comes up with doesn't fit the response options or the respondent just feels like his answer is not good enough, that it needs work before reporting it. There also might be differences between questions that ask about attitudes where we want a top of mind response versus questions about behavior where more searching of memory might ultimately come up with a more accurate answer.

As we talked I was reminded of how many of the metrics we've come to use in online surveys as measures of response quality are crude at best. We are quick to delete respondents who answer "too quickly" or straightline when there may well be circumstances under which those are perfectly reasonable response behaviors. But few people ever talk about getting rid of slow responders even though the quality of their answers may be as bad or worse than those who answer quickly. I think this points to our tendency to assess quality by using those things that are easily measured. We can count grids, measure straightlining, compute completion times, etc. but we don't have much of a clue about how to measure the impact of poorly formed questions, answer categories that don't resonate or questions that ask about things respondents just don't care about. My friend does his share of litigation consulting and sometimes he can convince a client to go through a systematic questionnaire development process that includes focus groups, cognitive interviews, and a pretest. He finds that when they do this the results they get with online panels are much better than when they just sort of wing it. But this sort of systematic questionnaire development process is not used as often as it should be. To our detriment for sure, but for respondents even more.

Can we really do two things at once?

Like most research companies mine now routinely includes cell phones in our telephone samples. Best practice requires that before we interview someone on a cell phone we determine if it's safe to do the interview. If, for example, the respondent is driving a car we don't do the interview. Yesterday someone asked me if it was ok to do the interview if the respondent is using a hands-free device. The research on this is pretty clear: the problem with cells phones and driving is the distraction, not the dexterity required to hold a phone in one hand and drive with the other.  There is no basis for making an exception for hands-free.

This reminded me that responding to survey questions is not easy; it takes some serious cognitive energy. Most researchers accept the four-step response process described a decade ago by Tourangeau, Rips and Rasinski:

1. Comprehension—understand the question and how to answer it (instructions)
2. Retrieval—search memory to form an answer
3. Judgment—assess completeness and relevance of the answer
4. Respond—map the response onto the right response category

When respondents execute this process faithfully we say they are engaged. When they short-circuit it we talk about lack of engagement. A person talking on a cell phone while driving can either drive or engage with the survey. It's a rare person who can do both well simultaneously.

Which brings us to one of my favorite subjects: respondent engagement and rich media (aka Flash) in Web surveys. What is the rationale for arguing that dressing a Web survey up with more color, pimped-up radio buttons, a slider bar, or a slick drag and drop answering device is going to encourage respondents to execute the four-step response process with greater care than if we just show them the same kind of standard screen they use to enter their credit card details on Amazon? Or are unfamiliar interfaces just a distraction that makes it even less likely? It might get someone to the next question, but is that enough?

Getting to the bottom of the respondent engagement problem

I've been working along with some colleagues on the lit review section of a paper for the ESOMAR Congress. The topic is "gamification" as the next experiment designed to increase respondent engagement in online surveys. As anyone who has done their homework knows the issue of survey respondent engagement did not arise with the growth of online panels and online surveys. Over 40 years ago two of the legends, Charlie Cannell and Rober Kahn, were arguing that there is an optimal length for a survey and once that length is exceeded respondents become less motivated to respond and put forth less cognitive effort, causing survey data quality to suffer. In 1981 Regula Herzog and Jerald Bachman identified the tendency for respondents to "straight-line" in large numbers of consecutive items that shared the same scale, especially as they progressed through the questionnaire. Ten years later Jon Krosnick introduced the term "satisficing" to describe the tendency for survey respondents to lose interest and become distracted or impatient as they progress through a survey, putting less and less effort into answering questions.

What I find especially striking about this "early" work is its tone. It's not accusatory. No fingers are pointed and there is no implication that these are "bad respondents." Reflecting its mostly psychological roots, the arguments say that when we create certain kinds of conditions with surveys this is how people will react. No one suggests that people who exhibit these kinds of behaviors don't deserve to be interviewed, that we need to get them out of our datasets or they don't deserve to be heard. This stands in stark contrast to how the same problem has been discussed over about the last five years in the context of online surveys. Name calling has been popular—inattentives, mental cheaters, speedsters, or simply, bad respondents. (Here I admit that I am as guilty as anyone of some this.) And in most circles, current rhetoric to the contrary, the emphasis still is almost totally on getting rid of people who exhibit these behaviors rather than seriously attacking what pretty much everyone who has studied these problems over the last half century agrees is the root cause: long surveys on not very interesting topics. And now to that double whammy the online paradigm offers almost no limits on how often you might be interviewed.

Another legend and former boss, Norman Bradburn, proposed a simple solution way back in 1977: convince respondents that the data are important. Unfortunately, we seem to have taken a different path.

HSRM – Day 2 AM

There is a bit of a cloud over the conference this morning. A number of the papers were written by and to be presented by government people who could not travel yesterday because of the potential government shutdown. So people from the contractor side are taking up the slack and delivering the government papers.

The morning session's topic is "Advances in Measuring Health Status and Behaviors." There are five papers and they focus mostly in two areas: physical and cognitive capacity in the elderly and various ways of measuring mental impairment. I'm not going to go paper by paper. In a couple of months you'll be able to download them all from the NCHS site. Instead I'll make a couple of observations.

The first is the systematic and thorough way in which these folks go about developing questions, questionnaires, and the procedures to administer them. I was especially amused by testing of methods to measure mental and physical capacity among the elderly! Cognitive interviewing, field tests, and sophisticated measurement models are all part of the toolbox. Those of us in commercial MR seldom get the time and the money to do this kind of serious questionnaire development. Our questionnaires are often downright bad. Some of that is lack of training but a lot of it is putting together questionnaires under incredibly tight deadlines, and sometimes questionnaires that will only be used once. But if we are serious about increasing respondent engagement questionnaire design is going to need a lot more focus than it's currently getting.

The second observation is that these folks may start with very long interviews (an hour is not unusual) but they still struggle with the same issues we do in MR: controlling questionnaire length in the face of the constantly expanding need for more data. So we heard about "planned missing data designs" that administer only part of a long battery of questions to any one respondent and then the imputation models to support a full analysis.