Let's ban the R word
The problem that won't go away


One of the links in the blog roll down on the right is to Jakob Nielsen's site, usit.com. Nielsen does a steady stream of Web usability research and has published a number of excellent books on the topic. It's always interesting to look at his work and see what lessons there might be in it for online questionnaire design. He has a new post this morning describing the usability problems people encounter when a Web site's user interface does not behave like they expect it to behave. He uses the term "mental models" to describe a user's beliefs about how a UI should perform. Those beliefs are formed from experience with the particular site and with other regularly-visited sites. Designers, being very creative people with a wide variety of skills, may have other mental models based in what is possible or their own sense of how a UI "should" behave. Things go awry when the user's mental model conflicts with the mental model the designer used when building the UI.

I find this easy to relate to because I still don't understand the mental model of the person or persons who designed the UI for Facebook. But I also have seen evidence of the problem in survey design. Take grids, for example. The standard grid has the attributes down the left side, the answer categories across the top, and the respondent is asked to select an answer for each row. There often is helpful little shading across alternating rows to keep the respondent on track. This works well because we westerners read left to right, down and left to right again. But sometimes for reasons good and bad we have tried a vertically oriented grid in which the respondent is asked to select one answer in each column. Even with helpful vertical shading, respondents struggle. They take longer to answer and in some cases give up altogether because the question does not fit their model of how a grid in a survey is supposed to work.

I have long believed that it's important that a Web survey behave like every other Web application that an Internet user encounters in his/her daily online interaction. When Amazon, Google, and even Facebook start using slider bars and drop and drag gadgets then we should, too. The empirical evidence suggests that these things have some advantage based in their novelty but they also slow people down and may even cause them to answer differently. Of course, it may be that the professional survey takers who have been the staple of online research have much different mental models than the rest of us. But as we try to expand on that base with broader recruitment across the Web through river sampling and similar techniques we likely will find that simple, straightforward, and consistent survey designs serve us best. Before you argue, have a look at this previous post describing what respondents prefer when given a choice.