Previous month:
June 2013
Next month:
August 2013

Posts from July 2013

Is research on research the real deal?

David Carr had a piece in last Sunday's New York Times about the difficulty of distinguishing journalism from activism. His first sentence sums up the issue pretty succinctly, "In a refracted media world where information comes from everywhere, the line between two 'isms' — journalism and activism — is becoming difficult to discern." His case in point is Glenn Greenwald, the Guardian reporter who has been breaking all the Snowden stories. Greenwald seems to be a guy with a very strong suspicion of government, and it shows in what he covers and how he writes about it. Cable news provides still better examples, where people who describe themselves as journalists routinely put a political agenda ahead of the objectivity that some of us expect (hope) from "the news."

We have a similar problem in market research when it comes to distinguishing between good methodological research and what we like to call research on research (RoR). Good methodological research is based in honest and objective scientific inquiry. Hypotheses are formed, the relevant literature reviewed, experiments designed and executed, data analyzed, hypotheses accepted or rejected, conclusions reached, and potential weaknesses in the research fully disclosed.  The best of these studies end up in peer-reviewed journals where they help us to build and refine research methods, brick by brick.

Much of RoR, on the other hand, has become something much different. It often features a point of view rather than a hypothesis, and the exploration of the data is a search for proof points ratBadscienceher an objective analysis aimed at uncovering what the data can tell us. The end product typically is a white paper, designed to sell rather than inform. We might attribute some of the poor quality of RoR to a lack of training and skill, but I expect most of it comes back to the simple fact that MR is a business. Academics achieve success by doing good solid research that earns the respect of their peers. MR companies succeed by selling more of their stuff.

All of which is not to say that there is not some good RoR being done, studies that are based in the fundamentals of objective scientific inquiry. It's just that it's getting harder and harder to tell the difference. And given the methodological disruption that has come to characterize our industry over the last decade, that's a real problem for all of us.