Previous month:
August 2006
Next month:
November 2006

Posts from October 2006

Are Those Long Surveys Worth It?

I've just read two papers with very similar designs and nearly identical conclusions. (One by Arthur Lugtigheid and Sandra Rathod for Survey Sampling and one by Mirta Galesic at the University of Maryland.)  Both researchers looked at three data quality indicators: time spent answering questions, the percent missing data or DK responses, and the number of characters keyed in open ends.  de Jong also looked at use of a special slider bar and Galesic looked at differentiation in grids, that is, degree of "straightlining."

Their results show significant changes in response behavior as respondents get further into the questionnaire:

  • They spend less time reading and answering questions.
  • They choose DK or other non-substantive responses more often.
  • They key fewer characters in open ends.
  • They are less likely to use gadgets like slider bars.
  • Their responses to questions in grids show less differentiation.

How long does it take before these behaviors evidence themselves?  Lugtgheid and Rathod say after about 20 minutes.

As an industry, we have tended to believe that if we can keep respondents on line, that is, achieve a relatively low termination rate, then the Web questionnaire is doing its job.  In an incentive driven mode like Web has become, respondents may finish but at what price?

Drop-Downs May Deliver Defective Data

For some time now Web survey designers have recognized that drop-downs are problematic, and so most avoid them.  Web usaiblity guru Jakob Neilsen, who has long railed against them, has once again spoken on the issue.  Below is his verbatim post:

Another reason to use drop-down menus sparingly, as I wrote in 2000:

Most users now have mice with scroll wheels. When you select a value in a drop-down and then use the wheel to move down the page, you often change the value of the drop-down selection instead. Users often don't realize that they have changed their input but proceed to submit the form.

'Nuff said.