A Literary Digest moment

Last week McKinsey released results from a study of the impact of US healthcare reform (The Affordable Care Act) on employer-provided health insurance. The report estimated that 78 million workers were likely to lose their employer provided health insurance once the law kicks in fully in 2014. This estimate is significantly at odds with other studies including those by the Congressional Budget Office, RAND and The Urban Institute, all of whom estimated an effect that was an order of magnitude smaller than McKinsey's. This was big news and anti-reform professional pols pounced on it. Others wondered about the differences and asked McKinsey to release the details of their methodology. At first McKinsey refused, claiming that the methodology was proprietary. A few days later they fessed up.

It turns out their "proprietary methodology" is one that many readers of this blog probably use all the time: a B2B online survey using an access panel. And while they "stand by the integrity and the methodology of the survey" they also note that comparing their results to those of the CBO and other studies is comparing "apples and oranges."  They didn't really mean to do that but understand "how the language in the article could lead the reader to think the research was a prediction, but it is not." Shame on those readers! Here is what McKinsey said in the article:

The Congressional Budget Office has estimated that only about 7 percent of employees currently covered by employer-sponsored insurance (ESI) will have to switch to subsidized-exchange policies in 2014. However, our early-2011 survey of more than 1,300 employers across industries, geographies, and employer sizes, as well as other proprietary research, found that reform will provoke a much greater response (p. 2).

I'm not so naïve that I don't understand that all McKinsey is doing here is promoting their consulting business. The bigger the problem seems the more urgent the need for high-priced consultants to fix it. Nonetheless, there are two important lessons for us as researchers. The first is old news, still another reminder that there is no scientific basis for how most of us practice online research. The risks of getting the wrong answer probably are greater with online B2B than with consumers. The second is the importance of asking David Smith's Killer Question Number 3: "Does the new incoming evidence square with your prior understanding of this subject?" In other words, do your results jibe with whatever else we know from other studies and other data sources? If they don't, then you have some digging to do. McKinsey knew their results contradicted what others had done with much more rigorous methodologies and then flaunted it rather than go back and take a closer look at what they were doing.

The question now is whether this cautionary tale will have any impact at all on how the public views the credibility of online or how we practice it. I would like to think so but that's probably wishful thinking.

Comments

[הדרכת הורים]

I like to read this article which is really mean to do but comprehend how the terminology in the post could head people to think the analysis was a forecast, but it is not.

[Reg Baker]

The point Doug raises above is an important one that I glazed over in this post. But I sort of spoke to it a year or so ago in this post: http://regbaker.typepad.com/regs_blog/2009/12/what-about-b2b.html

[Reg Baker]

"Remarkable" would be one word to describe it. This seems a little like the classic political scandal where the closer you look the worse it gets.

[doug r.]

Hi Reg,

Forgetting the sampling (this is disclosure??), the problem here is that they're screening a general population sample to find decisionmakers on health benefits. And then they imposed a quota on firm size AFTER screening. They claim to have found 223 primary decision makers in firms with more than 500 employees. In 2007 (the latest data from the establishment survey), there were 19,000 such firms in the US, to there are roughly 6 primary decision makers per 100,000 population. It would be quite remarkable if the Ipsos panel (of 600,000 persons, with something less than a 100% response rate) had 223 of these decision-makers.

I guess the moral is you shouldn't go to McKinsey looking for statistical expertise.

Cheers,

Doug

[twitter.com/duey23]

Sadly, I agree that it's wishful thinking. In the era of "news" demanding "leads if it bleeds" stories every single waking moment, this doesn't matter and they will use it to inflame (or calm) their main constituents to fit their own needs.