Nate Silver does a nice job this morning of summarizing the accuracy of and bias in the 2012 results of the 23 most prolific polling firms. I’ve copied his table below. Before we look at it we need to remember that there is more involved in these numbers than different sampling methods. The target population for most of these polls is likely voters and polling firms all have a secret sauce for filtering those folks into their surveys. Some of the error probably can be sourced to that step.
But to get back to the table, the first thing that struck me was the consistent Republican bias. The second was the especially poor performance by two of the most respected electoral polling brands, Mason-Dixon and Gallup. But my guess is that readers of this blog are going to look first at how the polls did by methodology. In that regard there is some good news for Internet methodologies, although we probably should not make too much of it.
As far back as the US elections of 2000 Harris Interactive showed that with the right adjustments online panels could perform as well as RDD. When the AAPOR Task Force on Online Panels (which I chaired) reviewed the broader literature on online panels we concluded this about their performance in electoral polling:
A number of publications have compared the accuracy of final pre-election polls forecasting election outcomes (Abate, 1998; Snell et al, 1999; Harris Interactive, 2004, 2008; Stirton and Robertson, 2005; Taylor, Bremer, Overmeyer, Sigeel, and Terhanian, 2001; Twyman, 2008; Vavreck and Rivers, 2008). In general, these publications document excellent accuracy of online nonprobability sample polls (with some notable exceptions), some instances of better accuracy in probability sample polls, and some instances of lower accuracy than probability sample polls. “ POQ 74:4, p.743
So there is an old news aspect to Nate’s analysis and one would hope that by 2012 the debate has moved on from the research parlor trick of predicting election outcomes to addressing the broader and more complicated problem of accurately measuring a larger set of attributes than the relatively straightforward question of whether people are going to vote for Candidate A or Candidate B. In Nate’s table there are nine firms with an average error of 2 points or less and four of the nine use an Internet methodology of some sort. I say “of some sort” because as best I can determine there are three methodologies at play. Two of the four (Google and Angus Reid) draw their samples to match population demographics (primarily age and gender). IPSOS, on the other hand, tries to calibrate its samples to using a combination of demographic, behavioral and attitudinal measures drawn from a variety of what it believes to be “high quality sources.” (YouGov, which is further down the list, does something similar.) RAND uses a probability-based method to recruit its panel. So there are a variety of methodologies at play in these numbers.
Back in 2007, Humphrey Taylor argued that the key to generating accurate estimates from online panels is understanding their biases and how to correct them. I tried to echo that point in a post about #twittersurvey a few weeks back. Ray Poynter commented on that post.
My feeling is that the breakthrough we need is more insight into when the reactions to a message or question are broadly homogeneous, and when it is heterogeneous . . . When most people think the same thing, the sample structure tends not to matter very much. . .However, when views, attitudes, beliefs differ we need to balance the sample, which means knowing something about the population. This is where Twitter and even online access panels create dangers.
I think Ray has said it pretty well.