It's been hard to go to an MR conference over the last couple of years and not get at least one presentation pitching the use of Flash as a cure all for much of what ills online—falling response rates, high abandonment, high levels of satisficing, etc. I have never found these presentations to be terribly convincing. It's hard to deny that many or perhaps even most online surveys are boring. I just have never been convinced that adding some color and some cool answering gadgets is the solution.
A more encouraging effort has come out of MarketTools and their SurveyScore product, an attempt to quantify the impact of survey design on respondent behavior. Their early research seemed to say that it's all about length but with a hint that screens with lots of grids also are a problem.
But I see some real ground being broken in an article by Andy Peytchev in the current issue of POQ. (Here I disclose that Andy is a former member of the UM team we have done considerable work with, that we collected the data he analyzed, and that he thanks me among others in his first footnote.) He starts by providing us with a framework for thinking about the problem of survey breakoff, itself a service given the state of the debate. And he tries to link that framework to the larger literature on nonresponse. Perhaps I will go into that in a later post, but here I want to focus on his results. His key finding is that respondents are most likely to breakoff when they encounter a page they perceive as burdensome. No surprise that one of the design elements that seems to drive breakoffs is grids. Others include:
- Section introductions where the shift to a new topic is a commitment some respondents are not willing to make
- Sensitive questions or those where judgment is required
- Open ends or, especially, screens with multiple open ends
- Multiple questions on a page
- Pages with novel respondent tasks, in his case, slider bars and constant sums/tallies
One of his more interesting findings is that respondents who breakoff generally are not satisificers. Rather they are people who are taking their time, seem to be reading and considering questions carefully, and in general putting forth good effort. There simply comes a time when they can't go on because the survey becomes too demanding. But he also notes that there is a relationship between likelihood of breakoff and formal education, suggesting that more challenging survey designs are especially difficult for those with less education.
This is an interesting line of research and I would hope to see more of this in the years ahead.