Cell phone data quality

My first taste of a methodological imbroglio was 25 years ago and involved the introduction of CAPI (computer-assisted personal interviewing). There was widespread speculation that interviewers using laptops for in-person interviewing might lead to unforeseen impacts on data quality. Empirical research taught us that we needn't worry and so CAPI became the standard.

More recently as the growth in wireless only households has made it necessary to include cell phones in our telephone samples there has been a lot of worry about the quality of data collected by cell. Poor audio quality, an increased likelihood for multitasking while being interviewed, and the possibility of environmental distractions are some of the things that people cite as possible causes for reduced data quality. Now research by Courtney Kennedy and Stephen Everett reported in the current issues of POQ has turned up little empirical evidence that such effects exist. They randomized respondents to be interviewed either by cell or landline and then looked at six data quality indicators—attention to question wording, straightlining, order effects, internal validity, length of open end responses, and item nonresponse. They found no significant differences in five of the six. The outlier was attention to question wording where they found some evidence that cell phone respondents may listen less carefully to complex questions than those responding by landline.

It's gratifying to know that there are researchers out there who approach new methods with skepticism and take a guilty-until-proven-innocent position. More gratifying still that other researchers do the hard work of carefully vetting those concerns with well-designed empirical studies.