The Survey Geek

A Literary Digest moment

Last week McKinsey released results from a study of the impact of US healthcare reform (The Affordable Care Act) on employer-provided health insurance. The report estimated that 78 million workers were likely to lose their employer provided health insurance once the law kicks in fully in 2014. This estimate is significantly at odds with other studies including those by the Congressional Budget Office, RAND and The Urban Institute, all of whom estimated an effect that was an order of magnitude smaller than McKinsey's. This was big news and anti-reform professional pols pounced on it. Others wondered about the differences and asked McKinsey to release the details of their methodology. At first McKinsey refused, claiming that the methodology was proprietary. A few days later they fessed up.

It turns out their "proprietary methodology" is one that many readers of this blog probably use all the time: a B2B online survey using an access panel. And while they "stand by the integrity and the methodology of the survey" they also note that comparing their results to those of the CBO and other studies is comparing "apples and oranges."  They didn't really mean to do that but understand "how the language in the article could lead the reader to think the research was a prediction, but it is not." Shame on those readers! Here is what McKinsey said in the article:

The Congressional Budget Office has estimated that only about 7 percent of employees currently covered by employer-sponsored insurance (ESI) will have to switch to subsidized-exchange policies in 2014. However, our early-2011 survey of more than 1,300 employers across industries, geographies, and employer sizes, as well as other proprietary research, found that reform will provoke a much greater response (p. 2).

I'm not so naïve that I don't understand that all McKinsey is doing here is promoting their consulting business. The bigger the problem seems the more urgent the need for high-priced consultants to fix it. Nonetheless, there are two important lessons for us as researchers. The first is old news, still another reminder that there is no scientific basis for how most of us practice online research. The risks of getting the wrong answer probably are greater with online B2B than with consumers. The second is the importance of asking David Smith's Killer Question Number 3: "Does the new incoming evidence square with your prior understanding of this subject?" In other words, do your results jibe with whatever else we know from other studies and other data sources? If they don't, then you have some digging to do. McKinsey knew their results contradicted what others had done with much more rigorous methodologies and then flaunted it rather than go back and take a closer look at what they were doing.

The question now is whether this cautionary tale will have any impact at all on how the public views the credibility of online or how we practice it. I would like to think so but that's probably wishful thinking.

Can we really do two things at once?

Like most research companies mine now routinely includes cell phones in our telephone samples. Best practice requires that before we interview someone on a cell phone we determine if it's safe to do the interview. If, for example, the respondent is driving a car we don't do the interview. Yesterday someone asked me if it was ok to do the interview if the respondent is using a hands-free device. The research on this is pretty clear: the problem with cells phones and driving is the distraction, not the dexterity required to hold a phone in one hand and drive with the other.  There is no basis for making an exception for hands-free.

This reminded me that responding to survey questions is not easy; it takes some serious cognitive energy. Most researchers accept the four-step response process described a decade ago by Tourangeau, Rips and Rasinski:

1. Comprehension—understand the question and how to answer it (instructions)
2. Retrieval—search memory to form an answer
3. Judgment—assess completeness and relevance of the answer
4. Respond—map the response onto the right response category

When respondents execute this process faithfully we say they are engaged. When they short-circuit it we talk about lack of engagement. A person talking on a cell phone while driving can either drive or engage with the survey. It's a rare person who can do both well simultaneously.

Which brings us to one of my favorite subjects: respondent engagement and rich media (aka Flash) in Web surveys. What is the rationale for arguing that dressing a Web survey up with more color, pimped-up radio buttons, a slider bar, or a slick drag and drop answering device is going to encourage respondents to execute the four-step response process with greater care than if we just show them the same kind of standard screen they use to enter their credit card details on Amazon? Or are unfamiliar interfaces just a distraction that makes it even less likely? It might get someone to the next question, but is that enough?

Zero defects: An admirable but elusive goal

Several years ago I was asked to write a chapter for a book called Methods for Testing and Evaluating Survey Questionnaires. So a couple of colleagues and I wrote something on testing online questionnaires. It led me to scratch the surface of the contemporary software testing literature where I learned that the industry had more or less run up the white flag on zero defects, that software has become so complicated and the competitive pressures to get releases out quickly so intense that most people had quietly given up on the idea of a first release being bug free. This struck me as somewhat analogous to what's happened in MR over the last decade: research designs have become more complex, the questionnaires to support them have followed suit, but the timelines clients insist on continue to get shorter and shorter. So questionnaires are more convoluted, there are more lines of code and more numbers to check but with less time to do it.

Of course, we all insist to our clients that we check it all and that remains the goal. But even if we have all the time we need there are two lessons I took from the software QA literature and they have to do with the priorities that should guide our approach to QA:

1. Focus first on the most important stuff, that is, the section of the questionnaire, the lines of code and the analytic outputs that will create the biggest problem if they are wrong.
2. Focus next on those places where there is most likely to be an error, that is, where the questionnaire or code is most complex and the numbers hardest to compute.

Then check everything else. Clients rightfully expect that every deliverable we give them be 100 percent correct. Getting there is not easy.

Sugging, Internet style

In the beginning we had Sugging and shortly thereafter came Frugging. (The correct capitalization on these terms has always eluded me.) Mercifully we strayed from this convention when we discovered push polls. The question before us: what should we call what newsmax.com is up to?

Here is the deal. They send out survey invitations to email lists of unknown origin ostensibly to get opinions about some current issue. If you click their link you actually get a choice of four or five surveys. Each one is five or six innocuous questions about Obama, Palin, Healthcare reform, etc. They have a standard intro that tells you that newsmax.com is a leading online news organization whose results are reported on all of the big media outlets, and the survey topic is always Urgent. (My favorite was that this was the first ever online poll conducted about Barack Obama.) At the bottom they ask for your email address, otherwise your "vote" is not counted. Give them your email address and the conservative propaganda starts flowing to your inbox.

But it doesn't stop there. As it turns out, they will accept a blank survey as long as you give them your email address. Then you get an offer for a free Dynamo World Band Emergency Radio if you will just subscribe to their magazine which they say features "exclusive stories the major media won't report." Doubling down on the paranoia they tell us that the Department of Homeland Security's advises that we all have a radio.

So it turns out that it's just Sugging after all.

Getting straight on response rates

AAPOR's Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys has long been the bible for survey researchers interested in systematically tracking response and nonresponse in surveys and summarizing those outcomes in standardized ways that help us judge the strengths and weaknesses of survey results. The first edition, published in 1998, built on earlier work by CASRO (1982), a document that seems to have disappeared into the mists of time. Since then the SD Committee within AAPOR, currently chaired by Tom Smith, has issued a continuing series of updates as new practices emerge and methods change.

A 2011 revision has just been released and a significant part of this revision is focused on Internet surveys. In doing so it makes an important point that is often overlooked in practice. That point is this: if we want to calculate a response rate for an Internet survey that uses an online panel, it's not enough to track the response of the sample drawn from that panel; we also must factor in the response to the panel's recruitment effort(s). This is relatively straightforward for the small number of panels that exclusively do probability-based recruitment (e.g., The Knowledge Panel or LISS Panel). But the vast majority of research done in the US and worldwide uses panels that are not recruited using probability methods. The recruitment methods for these panels vary widely but in almost all cases it's impossible to know with certainty how many people received or saw an invitation. And so, the denominator in the response rate calculation is unknown and therefore no response rate can be computed. (The probability of selection is also unknown which makes computation of weights a problem as well.)

For these reasons a "response rate" applied to nonprobability panels is uncalculable and inappropriate, unless the term is carefully redefined to mean something very different from its meaning in traditional methodologies. These also are the reasons why the two ISO standards covering market, opinion and social research (20252 and 26362) reserve the term "response rate" for probability-based methods and promote the use of the term "participation rate" for access panels, it being defined as "the number of respondents who have provided a usable response divided by the total number of initial personal initiation requesting participation." And, of course, all of this is also getting much more complicated as we increasingly move away from "classic" panels toward designs that look a lot like expanded versions of river sampling with complex routing and even repurposing of cooperative respondents.

To my mind all of this is symptomatic of a much larger problem, namely, the use of a set of tools and techniques developed for one sampling paradigm (probability) to evaluate results produced under a very different sampling paradigm (nonprobability). This should not surprise us. We understand the former pretty well, the latter hardly at all. But therein lies an opportunity, if we can leverage it.