Social media research absent the hype
It works!

Hoisted on their own petard

The current issue of Inside Research is out and it includes the mid-year update on online spending. Despite the headline that characterizes the growth as "soft" the numbers show the MR companies reporting to IR (and mine is one) estimating that their 2010 revenues from online will be around $2.2 billion, a 12 percent increase over 2009. By comparison, increases for 2009 and 2008 were a mere 3 percent and 2 percent respectively. Of course, a substantial amount of the 2010 growth is no doubt due to an improvement in MR bookings generally, so I guess you could say that 12 percent growth is soft when compared to 18 percent for 2007 and 21 percent for 2006.

In addition to the numbers the article has a series of anonymous quotes from reporting companies and one in particular caught my eye:

 "More and more clients shifted their data collection spending toward lower cost alternatives in '10. However, given the lower response rates that online data collection provides relative to other methodologies, we are finding clients reconsidering their decisions to make this move."

Uh? How can it be that there is anyone left in MR who believes that response rates from online panels have anything to do with quality or representivity? When you're working with a probability sample competently drawn from a high quality frame lower response rates immediately cause you to worry that representivity has been compromised. But with a panel, where representivity was sacrificed long ago, response rate is meaningless as a measure of quality. In fact, most sampling statisticians argue that you should not even bother to calculate it. If you do choose to calculate it then you need to do it at both the recruitment stage and the individual survey stage. When you do that, the numbers get really scary

Let's assume a US panel of two million adults. If we say that the US adult population is 200 million (it's actually a bit more) then by definition the response rate at the recruiting stage for the panel was around 1 percent. Now even if I get a whopping 50 percent response rate to my survey the total response rate is only about .5 percent (1 * .50). Yes, one-half of one percent.

There is an irony here. One of the arguments for moving to online back in the 1990s was that response rates in traditional methodologies had fallen so low (10 percent or less) that their results were no longer reliable. I'm sure you can finish the thought.