The Survey Geek

Thinking fast and slow in web survey design

I am a huge fan of Jakob Nielsen's work on web usability.  He has a post out this week--"Four Dangerous Navigation Approaches that Can Increase Cognitive Strain"--that puts web usability into a system 1/system 2 framework.  As I've said many times before, I believe that his research on web usaiblity has important implications for web survey design. 

In his post Nielsen offers evidence for a principle I have long aruged is important in web survey design: unfamiliar answering devices and complex instructions absorb cognitive energy and distract from the key task of simply providing an answer to a question. I'm not going to rehash Nielsen's full post here, but encurage you to follow the link and have a read for yourself.  You may want to pay special attention to dangerous navigation approach number four: "Fun" tools that become obstacles.

Faster is better

It’s the college basketball season and that means yours truly is spending way too much time in front of his TV.  One of the more annoying commercials that gets repeated over and over is this one by AT&T, driving home the message that faster is better.  At least on your iPhone.  It’s a sort of focus group with elementary school kids.  All staged, of course. It reminded me of the current buzz about System 1 and System 2 thinking, probably best described in Daniel Kahneman’s excellent book, Thinking Fast and Slow, and what that might tell us about what is a “good” survey question. 

Over about the last decade I had the good fortune to work with a group of old friends (who happen to be world-class survey methodolgists ) fielding experiments on web survey design.  The three of them have collaborated on a book to be published in April that pretty much sums up what they learned over the course of those experiments.   The experiments mostly consisted of varying the presentation of a set of questions to see what changes those different presentations produced in how people answered the questions.  One important variable analyzed in almost every case was response time.  Presentations that allowed respondents to answer quickly were generally judged to be better than those that took longer.

This sort of fits with what I think I have always known about good questionnaire design.  When we can design a question for which a respondent has a well formed and easily retrieved answer (System 1 thinking) we get good data, at least in the sense that the answer is what the respondent believes to be true and probably acts on.  But the more respondents have to think (System 2) the shakier it gets.  Or, in some cases, don’t bother to think at all.  Look no further than customer sat questionnaires and the difference between the top of mind opinion you get when you ask about overall satisfaction first (System 1 thinking again) versus when you ask it later after you taken the poor respondent through the full attribute set (i.e., forced System 2 thinking).  

The folks arguing for gamficiation of surveys questions seem to think that the longer someone takes to answer a question the better the answer.  While that may be true for certain types of questions, in most cases it’s probably a bad sign.  Faster probably really is better.

Measuring the right stuff

A few weeks back I saw a post by online usability specialist Jakob Nielsen titled, “User Satisfaction vs. Performance Metrics.”  His finding is pretty simple: Users generally prefer designs that are fast and easy to use, but satisfaction isn't 100% correlated with objective usability metrics.  Nielsen looked at results from about 300 usability tests in which he asked participants how satisfied they were with a design and compared that to some standard usability metrics measuring how well they performed a basic set of tasks using that design.  The correlation was around .5.  Not bad, but not great.  Digging deeper he finds that in about 30% of the studies participants either liked the design but performed poorly or did not like the design but performed well.

I immediately thought of the studies we’ve all seen promoting the use of flash objects and other gadgets in surveys by pointing to the high marks they get on satisfaction and enjoyment as evidence that these devices generate better data. The premise here is that these measures are proxies for engagement and that engaged respondents give us better data.  Well, maybe and maybe not.  Nielsen has offered us one data point.  There is another in the experiment we reported on here where we found that while the version of the survey with flash objects scored higher on enjoyment, respondents in that treatment showed evidence of lack of engagement at the same rate as those tortured with classic HTML.  They failed some classic traps at the same rate.

A cynic might say that at least some of the validation studies we see are more about marketing than survey science.  A more generous view might be that we are still finding our way when it comes to evaluating new methods.  Many of the early online evangelists argued that we could not trust telephone surveys any more because of problems with coverage (wireless substitution) and depressingly low response rates.  To prove that online was better they often conducted tests showing that online results were as good as what we were getting from telephone.  A few researchers figured out that to be convincing you needed a different point of comparison.  Election results were good for electoral polling and others compared their online results to data collected by non-survey means, such as censuses or administrative records.  But most didn’t.  Studies promoting mobile often argue for their validity by showing that their results match up well with online.  There seems to be a spiral here and not in a good direction.

The bottom line is that we need to think a lot harder about how to validate new data collection methods.  We need to measure the right things.

 

Norman doors and online questionnaire design

In usability circles the doors in the pictures on the right are called “Norman doors.”    Their named for Donald Norman who wrote a fascinating little book  called, The Design of Everyday Things. In the book Norman talks a great deal about affordances, that is, properties of objects that help us to do things with them.  Like open doors.  But the book is also about user-centered design, that is, the design of affordances in ways that make it easy for people to use them.  In Norman’s world, things are easiest when we already know how to use them, hardest when we have to read instructions to figure it out.  This distinction he calls “knowledge in the head” versus “knowledge in the world.”

To get back to those doors, often when we see handles like the ones in the pictures our instinct is to pull them. After all, that’s what handles are for.  That’s knowledge in the head at work.  The knowledge in the world says “Push,” but as often as not we don’t see that until the door doesn’t work like we expect.  If this has never happened to you go find a Norman door (they’re everywhere), sit down and watch people as they go in and out.

I sometime think of Norman and his doors when I try to do a survey that has slider bars, drag and drops and other forms of what their designers like to call “engaging designs.”  It seems to me that I spend as much time trying to figure out how to use some of the gadgets as I do coming up with answers to the questions. Worse yet, I see questionnaires that use different gadgets for the same question type. The survey becomes a puzzle to be solved. Yes, a game.  But it’s a game that sucks cognitive energy away from the real survey task, which is thinking about the subject matter and answering the questions.  I pull on that Norman door because I’m not thinking about how to open the door.  I’m thinking about something else. I instinctively pull the handle because that’s what handles are for. That is until the door doesn’t open.  Then my train of thought is interrupted while I solve the problem of how to open the door.  That's what happens when we encounter unfamiliar affordances.

The thing about those boring radio buttons and text boxes in surveys is that we all know how to use them because they are so ubiquitous on the Web.  For some of us even clicking on words like Comment, Reply or Follow falls in the same category.  It’s second nature to us.  Knowledge in the head at work.  But when is the last time you bought something online that required a slider bar?  We are used to clicking our state with a drop down and entering dates by popping up little calendars and clicking on the day, but how often do we drag something into a shopping cart?  Or sort things?  These devices rely on knowledge in the world.  We should not be surprised that “engaging designs” often produce different answers and take longer.

If we are going to have better online questionnaires we need to be sure to focus on the right things.  Make questionnaires easier, not harder.

Plus ça change – Part 2

Back in December I wrote the first of two posts noting the way in which some trends in our industry can rise and fall and then rise up again with all of the same promise, problems and fears. I noted the way in which the issues surround CRM and data mining seem to be having a second act in the big data movement. This is the second post and it's about the way in which the industry's drive to improve online questionnaires, knowingly or not, is drawing on some longstanding principles that until now have fallen out of favor.

One thing that most people in this industry seem to agree on is the sorry state of questionnaire design. We have redefined "good design" in terms of the things we can measure. So we count up the number of grids, page loads, words on the screen, etc. And at the end of surveys we ask the survivors how the experience they just plowed through compared to the vague benchmark of "other online surveys you have taken. " Back in the salad days of survey research we used the metaphor of conversation to describe a survey. The operative metaphor today seems to be closer to root canal.

But I see signs of hope. The first is what Jeffrey Henning has been blogging about under the heading of "Crowd-Shaped Surveys." The principle is simple: if you want questionnaires that work better for respondents then involve them in the design process. This principle is also a very old one. For decades researchers have used techniques such as focus groups, cognitive interviewing, think-aloud exercises, and pretests to fine tune survey questions so that they are easier for people to answer and yield the information that the researcher is looking for. These techniques are still widely practiced in academic and government circles but seldom used in MR. I guess we think that we just know how to write great questions. Jeffrey has been sharing a lot of ideas about how we can achieve some of the same goals via technology. I'm not crazy about all of what he suggests and look forward to seeing the validation studies, but it is one very promising path out of the rut we seem to be stuck in.

The second sign of hope is survey gamification, or at least some elements of how it is practiced by survey gamification guru Jon Puleston. (Jon also is an advocate of pretesting.) Now I admit that I'm no fan of Flash gadgets or many of the gaming features Jon has tested, but as I have written elsewhere one key element of Jon's approach is simply to write better questions by making them specific, creating a context, giving examples to clarify and trigger recall, etc. He reads the literature on questionnaire design and it shows in his work.

At the end of the day good questionnaire design is not about grids, page loads, interactivity, cute graphics or fancy answering devices. It's about using words well to ask questions that are clear and interesting to respondents and to provide a meaningful set of response options to capture the answers. That can be a tall order given some of the topics clients ask us to research, but there are ways to get there. It's encouraging to see MR remembering them.