Previous month:
October 2009
Next month:
December 2009

Posts from November 2009

More data on cell only households

While reports of the death of telephone interviewing continue to be premature it certainly is true that the double whammy of high rates of nonresponse and deterioration of the sample frame due to wireless substitution are serious problems in need of serious solutions. As I've noted in previous posts, current estimates put the percent of US cell only households at around 20 percent with perhaps an equal number of "cell mostly" households, meaning that they take most of their calls on their cell phone and therefore are very difficult to reach in the usual telephone survey design. Numbers like these have made it all but mandatory for a telephone research study to include calling cell phones if for no other reason than to maintain some level of face validity.

With this as backdrop Nancy Bates from the US Census Bureau has an interesting little piece out on Survey Practice. She has looked at the differences in preferences for online surveys versus mail surveys for households that are difficult to reach by telephone. While she finds only modest differences in Internet use across a continuum of cell only, cell mostly, cell sometimes, and landline only the preference for an online survey versus a mail survey across this same continuum is very striking. She reports that roughly 48 percent of cell only respondents report a preference for online versus just 13 percent for landline only.

Now there are multiple caveats here. She is asking about intent and it's not clear how behavior might differ. While I'm not going to try to dredge up the cites (you will just have to trust me) I have seen a handful of studies report that when given a choice between mail and Internet a significant plurality of Rs of all ages and stripes choose mail. In fact, we have a study going on right now that's driving home that very point. Nancy also is looking at filing our Census forms, not typical surveys that may be longer, more complex, and, frankly, less compelling. Finally, at some point we have to consider the possibility of mode effects using either of these approaches in a survey that also includes telephone interviewing.

Nonetheless, while I am offering lots of caveats I think this is a bit of research worth thinking about while we watch the telephone survey prognosis get darker and darker. But I also must confess that for the kind of work we do in MR simply calling cell phones may be the better path. Other sectors of the research industry may prefer another path.

If we knew what we were doing

At one point yesterday my wife showed me a wall plaque in a Signals catalogue with a quote from Einstein that read, “If we knew what we were doing it wouldn’t be called research.”  For some reason, it struck a chord.  And then this morning I was reading a review of the new Malcolm Gladwell book in the Times.   Now I confess that I pretty much loathe this whole genre of pop science (including other books like The Wisdom of Crowds and How We Decide) that has taken over the non-fiction side of the best seller lists all the while presenting itself as something we need to take really seriously.  And there seems to be no shortage of people in MR doing just that.  So when I read Steven Pinker’s review of  What the Dog Saw I literally jumped from my chair with glee.  He nails not just the Gladwell book but the entire genre when he describes the reasoning in Outliers as “cherry-picked anecdotes, post-hoc sophistry and false dichotomies.”  Sure, it’s entertaining stuff.  But it’s also lazy, shallow, and maybe even dishonest.  And then I thought about all that is being written and said about “the new MR” and I began to wonder whether we might be heading down this same road.  I hope not.

Einstein also said, “It’s hard.  That’s why we call it research.”

Let's hear it for conventional wisdom!

The post begins, "Contrary to conventional wisdom, U.S. consumers age 65 and older are most likely to shop online (77%). Consumers ages 45-64 and ages 30-44 follow at 71% with those ages 18-29 at 70%, a new report finds."  So 77%of persons aged 65 or older shop on line?  That's especially astounding because by most estimates only about 45% of persons 65 and older are online.  And presuming this study was done online (the post doesn't offer that detail) we need to also accept the reasoning that seniors on somebody's online panel are just like all online seniors in their use of the Internet.

Someone (The Cable and Telecommunications Association for Marketing) is actually selling this stuff!  One only hopes that nobody is buying it.

Clients say the darndest things

For the past couple of weeks I have been mulling Stan Sthanunathan's remark at the IIR conference in Las Vegas reported as "quality doesn't matter."  I've also been getting reactions from friends and colleagues around the industry.  They vary from "I'm glad we don't work for Coke" to "How should this impact our strategy" with a midpoint of "He's just trying to make a point." 

I'm not going to reprise his entire argument here. That's why there's a hyperlink in the first sentence.  But it's worth a read because it seems to me to raise an interesting question, one that I have asked here before.  Do we really understand what level of precision clients need for the business decisions they make? There is this tendency for researchers to believe that when a client commissions a piece of research the results of that research and that research only drives a business decision. And so it needs to be spot on in terms of measurement accuracy or you get a bad decision.   But when you talk to clients they often tell you that they look at a whole range of data points as part of their decision making process and that one study we did for them is only one piece of a much larger puzzle.

There is, of course, at least one other explanation.  That would be that clients expect us to deliver accurate results.  Those are table stakes.  We differentiate ourselves through interpretation, not measurement.  Either way, these are tough lessons for a survey geek. 

Clarification on the MR "exemption" to Do Not Call

A self-described "devoted reader" has sent me a private message reminding me that the status of the US MR industry relative to Federal Do Not Call legislation is not "exempted."  That would imply a specific exemption in the law which is incorrect.  MR simply is not included.   While operationally there's not much difference, the legal advantages of being not included as opposed to exempt are significant.  My son the attorney no doubt would be very disappointed in his old dad.  And thanks to the reader for pointing this out.