The Survey Geek

Cell only households continue to rise

CDC has just updated its estimate of cell only households from the National Health Interview Survey.  For those of you not paying attention, this is pretty much the gold standard for tracking the growth of cell only households.  CDC now reports that as of June of 2009, 22.7 percent of US homes have at least one cell phone but no landline.  Another 14.7 percent report having a landline but mostly using their cell phone(s) to place and receive calls.  Putting the two together, this means that a standard RDD telephone survey is likely to miss about 37 percent of households unless it is augmented to include cell phone sample.  While it adds time and expense, calling cell phones is rapidly becoming a standard feature of US telephone surveys.

I'd like to find a silver lining in this but can't.

More data on cell only households

While reports of the death of telephone interviewing continue to be premature it certainly is true that the double whammy of high rates of nonresponse and deterioration of the sample frame due to wireless substitution are serious problems in need of serious solutions. As I've noted in previous posts, current estimates put the percent of US cell only households at around 20 percent with perhaps an equal number of "cell mostly" households, meaning that they take most of their calls on their cell phone and therefore are very difficult to reach in the usual telephone survey design. Numbers like these have made it all but mandatory for a telephone research study to include calling cell phones if for no other reason than to maintain some level of face validity.

With this as backdrop Nancy Bates from the US Census Bureau has an interesting little piece out on Survey Practice. She has looked at the differences in preferences for online surveys versus mail surveys for households that are difficult to reach by telephone. While she finds only modest differences in Internet use across a continuum of cell only, cell mostly, cell sometimes, and landline only the preference for an online survey versus a mail survey across this same continuum is very striking. She reports that roughly 48 percent of cell only respondents report a preference for online versus just 13 percent for landline only.

Now there are multiple caveats here. She is asking about intent and it's not clear how behavior might differ. While I'm not going to try to dredge up the cites (you will just have to trust me) I have seen a handful of studies report that when given a choice between mail and Internet a significant plurality of Rs of all ages and stripes choose mail. In fact, we have a study going on right now that's driving home that very point. Nancy also is looking at filing our Census forms, not typical surveys that may be longer, more complex, and, frankly, less compelling. Finally, at some point we have to consider the possibility of mode effects using either of these approaches in a survey that also includes telephone interviewing.

Nonetheless, while I am offering lots of caveats I think this is a bit of research worth thinking about while we watch the telephone survey prognosis get darker and darker. But I also must confess that for the kind of work we do in MR simply calling cell phones may be the better path. Other sectors of the research industry may prefer another path.

No surprise: It just keeps getting worse

The latest numbers on wireless only households as measured by the NHIS have just been released.  They now estimate that as of December 2008 20.2 percent of US households have only a wireless telephone.  The 2.7 percent increase in the second half of 2008 is the largest six month increase since the NHIS has been collecting these data. 

I recently saw some estimates from SNL Financial that appear to have taken the NHIS trends and projected them forward.  Their data say that by 2013 a whopping 37 percent of US households will be wireless only. 

As I have noted in the past, the most statistically robust way of dealing with this problem is to spend the extra money and dial some cell phones.  The financial hit is not too bad as long as you only have to do limited dialing.  But as the proportion of cell only Rs increases, so do the costs.  While there is always the option of mixing in wireless only Rs from an online panel that is a journey into a statistical no man's land.

I don’t know

I have this assignment of sorts to read an often-cited article by Jon Krosnick and some colleagues titled, "The Impact of 'No Opinion' Response Options on Data Quality," Public Opinion Quarterly, 66:371-403. This is quite timely as I have just finished a bit of empirical research with some colleagues that cites this article, although I confess I have not read it in several years. The research was a Web-based experiment in which we tried to assess the impact of offering a DK option in a Web survey. This is of special interest because so much of Web is about transitioning phone studies to Web and understanding some of the differences one sees. Interviewers seldom read a DK option to respondents. Rather, they hold them in their pocket and use them as a sort of last resort. Online you need to decide whether to display the DK option or not, but when you do it's not unusual to get a significant increase in its use. Hardly surprising. The key research question is whether you get an overall different distribution of substantive responses depending on whether you offer a DK option. In our study (presented by Mick Couper at the General Online Research Conference in Vienna), we got significantly higher rates of nonresponse (i.e., more frequent selection of DK rather than simply skipping the question) when we presented it on the screen. Differences in the distribution of response items question by question were difficult to detect. So the takeaway here is that not presenting the DK reduces nonresponse and does not seem to lead to a lot of guessing and nonsensical answers that destabilize the distributions.

Krosnick and his colleagues come to essentially the same conclusion. There is a body of research (see, for example, Philip Converse, "Attitudes and Non-Attitudes: Continuation of a Dialogue, " in The Quantitative Analysis of Social Problems, ed. Edward Tufte, 1970) that argues sometimes people really don't have a "preconsolidated opinion" and you can't force them to come up with one. So to keep people from guessing or answering randomly, it's best to offer a DK. Krosnick and his colleagues argue that selecting the DK is a form of satisficing that is most prevalent among respondents with less formal education, in self-administered situations, and toward the end of the questionnaire. In other words, it's a sign that respondents are not willing to put forth the cognitive energy to give a thoughtful answer, so they jump at the chance the DK offers. Take away the option and these people mostly will give valid answers and you will reduce your nonresponse.

Of course, taking away the DK option will not drive nonresponse to zero. Survey research, like life, is full of compromise. Unfortunately, due to a design flaw introduced by yours truly our research could shed no light on what happens if you don't offer a DK and don't let respondents just skip the question. On many commercial Web surveys there is no DK and an answer is required, which may produce a different result. That's a variation we need to test, but at least in this experiment our results were not terribly different from what Krosnick describes.

More comparisons of Web to other methods

I am finally getting around to wading through the mother lode of academic research noted in an earlier post way back at the beginning of March.  The special POQ issue has two articles, one looking at Web versus face-to-face and the other comparing CATI, Web and IVR.  The results are not particularly surprising, but it's nice to see one's suspicions confirmed with well-designed and executed research.

Dirk Heerwegh and Geert Loosveldt report on results from a survey in Belgium designed to assess attitudes toward immigrants and asylum seekers. They put considerable effort into designing both the Web and face-to-face survey based in Dillman's unimode construction principles.  In other words, they worked hard at making the two surveys as comparable as possible rather than optimizing each to its own mode.  Their results are pretty convincing.  The Web survey produced a higher rate of "don't know" responses, more missing data, and less differentiation in scales.

Frauke Kreuter, Stanley Presser, and Roger Tourangeau looked at social desirability bias across three methods--one with an interviewer (CATI), one without an interviewer (Web), and one with sort of an interviewer (IVR).  They drew a sample of University of Maryland alumni and asked a variety of questions about academic performance and post graduation giving.  They were able to verify the respondent's answers against university records.  In essence, they were able to tell who was telling the truth and who was not.  As with Heerwegh and Loosveldt, the results are pretty much what we would expect.  Web reporting was the most accurate and CATI the least with IVR generally somewhere in the middle.

So there you have it.  We used to like to say that "the Internet changes everything."  Well, it does not appear to have changed some basic principles of survey research.