The Survey Geek

Practice makes perfect

This is something of an unplanned follow-up to my last post.  While catching up on my reading I came across an interesting article in the winter issue of POQ by LinChiat Chang and Jon Krosnick reporting on a neat little study that compares responses from a telephone RDD sample, an online panel recruited by RDD (Knowledge Networks), and an online volunteer panel (Harris Interactive).  One key finding is in sync with something I have heard talked about in connection with the ARF study: online panel respondents do a good job of completing questionnaires and the more they do the better they get. 

As a broad summary, in the Chang and Krosnick study the online panel results showed less satisifcing, less social desirability, better self-reports, and greater internal consistency than did the results from the RDD telephone sample.  The online panel folks were just better at doing surveys. Some of that was attributed to more practice, but some of it also was due to a stronger tendency for the volunteer panel respondents to select only those studies where they had a strong interest in the survey topic.  The telephone results, on the other hand, were more representative demographically and in terms of electoral participation.  These differences persisted even after weighting.

A cynic might say, "Pick your poison."  An optimist might say, "Good input to fit-for-purpose methodology choices."  I might wonder whether we've used up a lot of time, money, and stomach lining worrying about the wrong problem.

Just a little whining

A couple of weeks back Research asked a number of industry wise men and women to pick one word to sum up MR in 2010.    Jeffrey Henning chose “probability.”  He goes on to say, “What ‘subprime lending’ was to the financial industry, ‘access panel’ was to the market research industry.”   He then references the Yeager and Krosnick study released in 2009 showing that probability samples still trump convenience samples  when it comes to accurate measurement.  Just above Jeffrey’s commentary was Ray Pointer’s, and in Ray’s commentary he wrote, “the industry has largely abandoned random probability sampling.”  So who is right?

When the online panel chickens first came home to roost three or four years ago and the great panel data quality crisis ensued I, much like Jeffrey, predicted “a flight to quality.”  It never happened.  The industry managed somehow to pivot on the issues and to redefine quality into what MarketTools has neatly summed up with their TrueSample tagline, “Real, Unique, and Engaged.”   Tired old concepts such as representivity and accuracy were defined out of the equation.  Survey results need only be consistent and directional.  The most important characteristic of a survey respondent became the willingness to faithfully complete our long, tedious and boring surveys, rather than the natural ability to accurately represent everyone else out there sharing the same attitudes, beliefs, behaviors and views of the world.

There are lots of reasons why this has happened and why it’s probably never going back.  Chief among them is the fact that our customers no longer place any real value on accuracy.  It’s just too boring.  Clients want “insight.”  Intuition and gut feelings are valued above plodding, systematic analysis.  Pop science has driven out real science.  Anecdotal evidence is sufficient. 

I want to write a paper titled, “The End of Representivity: Market Research in the Age of Blink.”

So, yes, I think Ray has it right.  But he went on to say that researchers will “need to get their heads around concepts such as triangulation and confirmatory, disconfirming, and maximum variation samples.”  Right again.  But I wouldn’t count on it.

One bad tweet

Earlier in the week a friend forwarded on to me that silly one bad Twitter tweet post from last November claiming that a negative tweet about a brand causes it to lose 30 customers.  I wish I had that kind of power because there are some companies out there I'd like to cause some major hurt!  That same day I got another email that eventually sent me to this piece of research by the WOM experts at Keller Fay Group.  I am embarrassed that I missed it given its ESOMAR award and its interesting perspective on WOM and social media.  Key findings from the study:

• Ninety plus percent of word of mouth conversations take place offline with just seven percent online, and the overwhelming majority of those online conversations are either by email or IM, not blog posts.
• Offline conversations about brands are felt to me more credible and more likely to lead to purchase.
• Half of all online conversations about brands are among teenagers.

In truth, these estimates probably are even worse news for social media than they seem.  The study was done in the US and online, so when you consider that about one in four Americans don't even use the Internet and the tendency for online respondents to be among the heaviest Internet users that seven percent starts to look like overkill.

I probably am more prone to embrace technology hype than the average guy, but this makes me think twice about just how soon social media will be the research bonanza we've been imagining.