The Survey Geek

Are Drop Downs Finally Down for the Count?

For some while now I have advocated against the use of drop downs in Web surveys.  That's been based on a number of things--some research we've done with our UM colleagues, an emerging consensus in the usability literature, and my own personal experience. 

Benjamin Healey from Massey University in New Zealand has just provided what I hope is the last nail in this coffin.  He tested drop downs against standard radio buttons and has published his results in in the current issue of Social Science Computer Review ("Drop Downs and Scroll Mice: The Effect of Response Option Format and Input Mechanism Employed on Data Quality in Web Surveys.").  His findings are straightforward:

• Drop downs produce higher levels of item nonresponse than radio buttons.
• It takes respondents longer to answer using a drop down than with a conventional radio button.
• A majority of respondents (76 percent)  using scroll mice accidentally changed an answer at least once.
• Respondents in the drop down condition were more likely to choose answers toward the bottom of the answer set than those with radio buttons.

So higher respondent burden and lower quality data.  It makes you wonder why we continue to see these things in surveys.

Behind Those Low Response Rates

I just had reason to go back and take another look at an article that appeared in Public Opinion Quarterly back in 2005.  It was written by three researchers at UM (Richard Curtin, Stanley Presser, and Eleanor Singer) and looks in some detail at the declining response on one of the most well known telephone surveys: The Survey of Consumer Attitudes.  It also is a pretty rigorous survey--lots of calls (at least six tries on no answers and rotating through calling windows.)

The article charts the fall in response rate from 72 percent in 1979 to 48 percent in 2003.  It describes two distinct periods of decline: about a .75 point per year decline between 1979 and 1996 and a more severe decline of about 1.5 points per year between 1997 and 2003.  (The rumor is that it's gotten much worse, although I don't have specifics on that so please don't quote me.) 

More interesting, however, is the source of the low response rate.  The refusal rate has increased from 19 percent in 1979 to 27 percent in 2003, although it was fairly level at around 20 percent until around 1999 and then shot up significantly.  The bigger story is the increase in the noncontact rate from about 3 percent in 1979 to 17 percent by 2003.

The authors blame a combination of telemarketing and the proliferation of surveys, both of which have driven people to answering machines, caller ID, and just plan old not answering the phone.  These no doubt are major issues, but there may be another problem here and it has to do with changing lifestyles.  People are not home like they used to be--especially younger adults--and cell phones are rapidly replacing landlines as the primary means of communication.  Things like the Do Not Call Registry and use of better respondent cooperation techniques might help with the refusal problem, but lifestyle changes are a tough order.