The Survey Geek

When will we learn?

I am delighted to see Mirta Galesic's and Michael Bosnjak's 2003 study of the impact of questionnaire length on response quality in Web surveys finally make it into print, in this case the summer issue of POQ. We all believe down to our toes that long surveys are bad, but I have been disappointed on more than one occasion to see how poorly prepared we sometimes are to make the arguments against length to clients who too often want surveys that stretch beyond what we believe to be reasonable.

Mirta and Michael help to fill that gap in a well-designed experiment. I'm not going to describe the details of the design but just go right to key finding: As survey length and complexity increase response quality declines. Respondents spend less time answering individual questions, skip questions or choose non-substantive responses (like DK) more often, key fewer characters in open ends, show less response differentiation in grid-style questions, and are more likely to quit the survey completely. These behaviors become especially prevalent at around 20 minutes.

Many studies since, have replicated these findings. I expect few readers of the article will be surprised by what they see. Google 'questionnaire length and response quality' and you'll see what I mean.

The obvious question: since there is all of this empirical evidence why does survey length continue to be such a problem?

 

.

More thumbs down on eye candy in surveys

The current issue of Inside Research has a short note on some work done at DMS looking at respondent reactions to rich text and graphical interfaces in Web surveys. It's hard to go to a MR conference these days without seeing someone present a passionate argument for the use of Flash or various kinds of interactive gadgets as a way to increase respondent engagement and therefore improve data quality. Readers of this blog know I am skeptical. Nonetheless, I tracked down the presentation from Chuck Miller at DMS.

It turns out to be an interesting piece of research. (There is a video of Chuck describing his research here.) He took sample from a variety of sample sources and allocated it out to three treatments: (1) forced into a plain text survey; (2) forced into a rich text survey; and (3) allowed to choose either plain text or forced text. Quotas were used to force the demographic distributions. The key findings:

• Three out of four respondents in the choice treatment chose plain text.
• Choice seemed to be driven by lifestyle (e.g., lots of time online, electronic gadget geek, etc.) rather than demographics. Rs choosing rich text also reported taking lots of surveys.
• Rs in the rich text group gave higher ratings of the survey experience than those in the plain text group, but mid-survey terminations in the forced rich text group were higher than the forced plain text group.
• For the most part, there were no differences in response data itself except for the slider bar questions where Rs using the slider bars reported significantly more time watching TV, listening to radio, and reading magazines than those using standard radio buttons.
• There were no significant differences across a range of data quality indicators except for straightlining which was higher in the plain text treatment. However, I have always felt that the standard radio button grid lends itself to straightling while many of the rich media answering devices do not, but that doesn't mean the latter is yielding more thoughtful answers.
• Overall, the best approach seems to be to let people choose their interface, but the practicality of that with today's software tools is questionable.

I am reminded of research, now several years old, that looked at slider bars and sorting gadgets back when we first figured out how to incorporate them into surveys. That research mostly reported that these devices caused us to lose more respondents (either because of technical issues, confusion, or not liking the device), generally took longer to answer, and produced the same results as conventional answering devices.

This whole debate, if there is one, is symptomatic of a basic problem in the evolution of online research. Someone gets an idea, sometimes builds a business around it, and advocates it loud and far as the next big thing with no real data or facts to substantiate the claims. Because it's cool or fast or less expensive we buy into it, only to eventually find an emperor with no clothes.

We really need to do better.

Well said

I'm catching up with some reading on a flight to DC and have come across an interesting interview in the May issue of Research World. While interviewing Obama's pollster, Joel Benenson, Simon Chadwick asks, "Is it all about how you analyze the data?" Benenson's response is especially eloquent:

"Look, if you write a good survey, you have written a survey to produce data that's going to be impactful in terms of your analysis and in terms of the decisions that you're trying to make. You have some hypotheses when you draft the survey, and you develop the best way to test those hypotheses. I view the questionnaire as an art form. That's where you really lay down the architecture of what your data could look like, and if you've done it well you're going to have a very rich data set."

Just yesterday I turned down an opportunity to be interviewed by someone about DIY surveys. As I said at the time, "I don't have any data to show how bad they are; mostly I just emote about it." My argument against them generally comes down to this: well-trained researchers add value because they know how to write good questions and good questionnaires. Benenson has said it so much better.

Unfortunately, it's not just about DIY. The MR industry has become so fixated on the analytics on the back end that we seem to have lost sight of the importance of how we get the data to begin with. Simon's question reflects that bias; Benenson's answer sets us straight.

Sometimes I am slow to catch on

I've been on vacation and literally the last thing I did before I left was to read a piece in IJMR by Martin Oxley and Brendan Light titled, "Research 2.0: engage or give up the ghost?"  As I mulled it over on the way to the airport it dawned on me that this mixed bag of issues, ideas, and problems we lump under the general heading of "online" has four distinct themes to it.

The first of these themes is related to how we adopt traditional question and answer formats to the Web.  Mostly this is about how we transition from interviewer administration to self administration and from aural to visual.  This is the space where the academic-based survey methodologists have done a lot of good work with some worthwhile contributions from MR research on research.  This is also the turf of the Flash evangelists who I cynically view as more about promoting a business model than doing good solid methodological research.

The second theme is really a major paradigm shift and is all about changing how we extract (if that's the proper word) the information we need from people.  Rather than asking respondents (again, if that's the right word) questions that require them to translate their reactions, thoughts, opinions, etc. into words it tires to provide images and symbols as the main answering devices.  That's the essence of what Oxley and Light describe.  These methods have generally been more widely promoted and used in Europe than elsewhere.

The third theme is focused on the people whom we interview (again, if that's the right word).  This is where we get into all of those arguments about representivity, professional respondents, satisifcing, etc.  There probably has been more work in this area than the others, especially in the US, but how much light is being shed remains an open question.  This has been mostly about access panels but increasingly now is about social networks. Rightly or wrongly, this has been the turf where the debate about the quality of online research has been joined.

Finally, there is the "listening" theme.  This is all about harvesting the ongoing conversation of the Web to help clients understand how their products and brands are being perceived.  Too often it's not just about listening, but also about intervening which creates an interesting dilemma for researchers around where research ends and marketing begins.

Maybe this has been obvious to other people all along, but it's been a revelation of sorts to me.  I always find it easier to understand a problem when you can compartmentalize it and look at its pieces one at a time.

Some light shed on Web survey breakoffs

It's been hard to go to an MR conference over the last couple of years and not get at least one presentation pitching the use of Flash as a cure all for much of what ills online—falling response rates, high abandonment, high levels of satisficing, etc. I have never found these presentations to be terribly convincing. It's hard to deny that many or perhaps even most online surveys are boring. I just have never been convinced that adding some color and some cool answering gadgets is the solution.

A more encouraging effort has come out of MarketTools and their SurveyScore product, an attempt to quantify the impact of survey design on respondent behavior. Their early research seemed to say that it's all about length but with a hint that screens with lots of grids also are a problem.

But I see some real ground being broken in an article by Andy Peytchev in the current issue of POQ. (Here I disclose that Andy is a former member of the UM team we have done considerable work with, that we collected the data he analyzed, and that he thanks me among others in his first footnote.) He starts by providing us with a framework for thinking about the problem of survey breakoff, itself a service given the state of the debate. And he tries to link that framework to the larger literature on nonresponse. Perhaps I will go into that in a later post, but here I want to focus on his results. His key finding is that respondents are most likely to breakoff when they encounter a page they perceive as burdensome. No surprise that one of the design elements that seems to drive breakoffs is grids. Others include:

• Section introductions where the shift to a new topic is a commitment some respondents are not willing to make
• Sensitive questions or those where judgment is required
• Open ends or, especially, screens with multiple open ends
• Multiple questions on a page
• Pages with novel respondent tasks, in his case, slider bars and constant sums/tallies

One of his more interesting findings is that respondents who breakoff generally are not satisificers. Rather they are people who are taking their time, seem to be reading and considering questions carefully, and in general putting forth good effort. There simply comes a time when they can't go on because the survey becomes too demanding. But he also notes that there is a relationship between likelihood of breakoff and formal education, suggesting that more challenging survey designs are especially difficult for those with less education.

This is an interesting line of research and I would hope to see more of this in the years ahead.