The Survey Geek

There you go again!

One hears lots of silly things said at MR conferences and one of the silliest and oft-repeated refrains is that you can't do surveys with probability samples any more. There are even those who say that you never could. As often as I get the chance I point out that that's total nonsense. Lots of very serious organizations draw high quality probability samples all the time and get very good results. The prime example here in the US is the Current Population Survey, the government survey used as the basis for calculating the unemployment rate each month. Pretty much everything that comes from Pew is based in probability sample surveys as are many of those political polls that we follow so breathlessly every four years.

The concept of a probability sample is very straightforward. The standard definition is a sample for which all members of the frame population have a known, nonzero chance of selection. Unless you have a complex stratified or multi-stage design it's a pretty simple concept. As long as you have a full list of the population of interest to draw on and everyone has a chance to be selected the resulting sample can be said to represent the population of interest. But there are some serious challenges in current practice.

The first is assembling a frame that includes the entire population you want to study. For example, because of the rise of cell phone only households the landline frame that used to contain the phone numbers of well over 90% of US households no longer does. So it has become standard practice to augment the landline frame with a cell phone frame to ensure full coverage of the population. Clients often can supply customer lists that do a good job of covering their full customer base and we can draw good samples from them as well. Online panels are problematic because they use the panel as the frame and it contains only a very small fraction of the total population.

The second major challenge is declining cooperation. While there are studies that show even surveys with alarmingly low response rates can produce accurate estimates, low response rates make everyone nervous, raise fears of representivity and call results into question. The Current Population Survey gets 90% plus and so we trust the employment rate, but that kind of response is very unusual.

There are other challenges as well but I think it's the deterioration of the landline frame and very low response rates that cause some people to think that probability sampling is no longer possible. Anyone willing to spend the time and the money will get very accurate estimates from a probability sample, better than anything they'll get with an online panel or other convenience samples.

As I have written numerous times on this blog, the lure of online has always been that it's fast and cheap, not that it's better. And depending on how the results are to be used the method can be just fine, fit for purpose. But sometimes the problem requires representivity and when it does probability sampling is still the best way to get it.

A Literary Digest moment

Last week McKinsey released results from a study of the impact of US healthcare reform (The Affordable Care Act) on employer-provided health insurance. The report estimated that 78 million workers were likely to lose their employer provided health insurance once the law kicks in fully in 2014. This estimate is significantly at odds with other studies including those by the Congressional Budget Office, RAND and The Urban Institute, all of whom estimated an effect that was an order of magnitude smaller than McKinsey's. This was big news and anti-reform professional pols pounced on it. Others wondered about the differences and asked McKinsey to release the details of their methodology. At first McKinsey refused, claiming that the methodology was proprietary. A few days later they fessed up.

It turns out their "proprietary methodology" is one that many readers of this blog probably use all the time: a B2B online survey using an access panel. And while they "stand by the integrity and the methodology of the survey" they also note that comparing their results to those of the CBO and other studies is comparing "apples and oranges."  They didn't really mean to do that but understand "how the language in the article could lead the reader to think the research was a prediction, but it is not." Shame on those readers! Here is what McKinsey said in the article:

The Congressional Budget Office has estimated that only about 7 percent of employees currently covered by employer-sponsored insurance (ESI) will have to switch to subsidized-exchange policies in 2014. However, our early-2011 survey of more than 1,300 employers across industries, geographies, and employer sizes, as well as other proprietary research, found that reform will provoke a much greater response (p. 2).

I'm not so naïve that I don't understand that all McKinsey is doing here is promoting their consulting business. The bigger the problem seems the more urgent the need for high-priced consultants to fix it. Nonetheless, there are two important lessons for us as researchers. The first is old news, still another reminder that there is no scientific basis for how most of us practice online research. The risks of getting the wrong answer probably are greater with online B2B than with consumers. The second is the importance of asking David Smith's Killer Question Number 3: "Does the new incoming evidence square with your prior understanding of this subject?" In other words, do your results jibe with whatever else we know from other studies and other data sources? If they don't, then you have some digging to do. McKinsey knew their results contradicted what others had done with much more rigorous methodologies and then flaunted it rather than go back and take a closer look at what they were doing.

The question now is whether this cautionary tale will have any impact at all on how the public views the credibility of online or how we practice it. I would like to think so but that's probably wishful thinking.

Vendors say the darndest things

Today I got a promo piece from a global online panel vendor. On the front page they describe their offering as "high quality representative and validated panels in Brazil, Russia, India, China and now Mexico."  According to Internet World Stats the Internet penetration in these countries is 38%, 43%, 7%, 32%, and 27% respectively.  I renew my plea: let's ban the R word.

Let's focus on the third leg

Kristin Luck writes a column for RBR and for two issues in a row she's worked the worry beads about what she sees as declining interest in online panel data quality. In the current column Kristin draws heavily on a reaction to her first column by Jackie Lorch. Jackie summarizes two legs of the online data quality stool (respondent validation and better survey design). Kristin points out that there is still a problem, quoting an anonymous client who says, "There hasn't been the progress with online research that many in the industry think. We have internal tests that clearly demonstrate that."

As I have written in this space before, respondent validation and better survey design while necessary are not sufficient to really fix the problem with these sample sources. We also must address the third leg of the stool: the selection bias built into pretty much every online panel and intercept method. Every time the issue of online panel data quality comes up that graph from the first stage of reporting of the ARF Foundations of Quality Initiative showing all of the variation in results across the 17 panels   pops into my head and I ask myself whether improved respondent validation and better survey design can make this problem go away. I think not.

We need a lot more focus on the third leg of the stool and there are encouraging signs. More and more researchers are concluding that simple weighting by demographics is not enough. Propensity weighting has been widely discussed although not broadly used, and while evaluations of it in academic journals have not been all that positive additional experimentation could produce a breakthrough. More promising, I think, is experimentation with new approaches that focus on the sample selection stage and the use of high quality probability surveys (.e.g, the American Community Survey, the General Social Survey) as a reference point for selecting representative samples from single or multiple panels.

Kristin may be worried but I am more optimistic than at any time since online panels first came on the scene almost 15 years ago.

Let’s get on with it

I spent some time over the weekend putting the finishing touches on a presentation for later this week in Washington at a workshop being put on by the Committee on National Statistics of the National Research Council. The workshop is part of a larger effort to develop a new agenda for research into social science data collections. My topic is "Nonresponse in Online Panel Surveys." Others will talk about nonresponse in telephone surveys and in self-administered surveys generally (presumably mail). The workshop is part of an overall effort driven by the increasing realization in the scientific side of the industry that as response rates continue to fall a key requirement of the probability sampling paradigm is violated. And so the question becomes: what are we going to do about it?

My message for this group is that online panels as we have used them in MR so far are not the answer. As I've noted in previous posts, response rates for online panels typically are an order of magnitude worse than telephone. At least with the telephone you start out with a good sample. (Wireless substitution is a bit of a red herring and completely manageable in the US.) With online panels you start out with something best described as a dog's breakfast. While it's become standard practice to do simple purposive sampling to create a demographically balanced sample that's generally not enough. To their credit, Gordon Black and George Terhanian recognized that fact over a decade ago when they argued for "sophisticated weighting processes" that essentially came down to attitudinal weighting to supplement demographic weighting to correct for biases in online samples. But understanding those biases and which ones are important given a specific topic and target population is not easy, and it doesn't always work. So a dozen years and $14 billion of online research later the industry seems to be just weighting online samples by the demographics and stamping them "REPRESENTATIVE."

The facts would seem to be these. On the one hand, you can still draw a terrific probability sample but the vast majority of people in that sample will not cooperate unless you make the extraordinary and expensive efforts that only governments have the resources to make. On the other hand, online panels have demonstrated that there are millions of people who not only are willing but sometimes eager to do surveys, yet we've not developed a good science-based approach for taking advantage. I take hope in the fact that some people are at least working on the problem. Doug Rivers regularly shares his research on sample matching which is interesting, although I've not seen applications outside of electoral polling. GMI's new Pinnacle product also is interesting, but so far I've only seen a brochure. And statisticians tell me that there is work in nonprobability sampling in other fields that might be adapted to the panel problem.

My message to the workshop group this week is simple: "Let's get on with it."