Just a little whining
January 25, 2010
A couple of weeks back Research asked a number of industry wise men and women to pick one word to sum up MR in 2010. Jeffrey Henning chose “probability.” He goes on to say, “What ‘subprime lending’ was to the financial industry, ‘access panel’ was to the market research industry.” He then references the Yeager and Krosnick study released in 2009 showing that probability samples still trump convenience samples when it comes to accurate measurement. Just above Jeffrey’s commentary was Ray Pointer’s, and in Ray’s commentary he wrote, “the industry has largely abandoned random probability sampling.” So who is right?
When the online panel chickens first came home to roost three or four years ago and the great panel data quality crisis ensued I, much like Jeffrey, predicted “a flight to quality.” It never happened. The industry managed somehow to pivot on the issues and to redefine quality into what MarketTools has neatly summed up with their TrueSample tagline, “Real, Unique, and Engaged.” Tired old concepts such as representivity and accuracy were defined out of the equation. Survey results need only be consistent and directional. The most important characteristic of a survey respondent became the willingness to faithfully complete our long, tedious and boring surveys, rather than the natural ability to accurately represent everyone else out there sharing the same attitudes, beliefs, behaviors and views of the world.
There are lots of reasons why this has happened and why it’s probably never going back. Chief among them is the fact that our customers no longer place any real value on accuracy. It’s just too boring. Clients want “insight.” Intuition and gut feelings are valued above plodding, systematic analysis. Pop science has driven out real science. Anecdotal evidence is sufficient.
I want to write a paper titled, “The End of Representivity: Market Research in the Age of Blink.”
So, yes, I think Ray has it right. But he went on to say that researchers will “need to get their heads around concepts such as triangulation and confirmatory, disconfirming, and maximum variation samples.” Right again. But I wouldn’t count on it.