Twitter and electoral polling: Not ready for prime time

Last night I chaired a panel at PAPOR on the general subject of social media and the 2012 election. The panelists were Paul Hitlin from PEW and Mark Mellman of the Mellman Group. The topic quickly narrowed to Twitter and the 2012 election. Both panelists had done substantial tracking of Twitter over the course of the campaign and showed multiple comparisons of Twitter volume, topics, and sentiment to polling results.

There were two main conclusions. Not surprisingly, while there were points of agreement between Twitter and the polls it was often weak, and at other times the two sources moved in opposite directions. Most striking was a Pew comparison showing a Romney surge on Twitter in the last week that seemed to say the he would win. (Hitlin started his presentation by claiming that if Twitter had been correct Ron Paul would be president!)

The second conclusion was more interesting. While the presenters agreed that Twitter did a poor job of tracking the campaign there were numerous disagreements between them, most of which they expected were rooted in their having used different software. Hitlin used Crimson Hexagon while Mellman used the Twitter-approved Topsy. So the results of both projects clearly demonstrated the degree to which the choice of software matters and the unpreparedness of many researchers at this point to make informed choices. No one in the room, including me, was prepared to debate the choice of one approach over the other. ESOMAR has tried to provide some help with its 24 Questions, but while we might know the questions to ask most of us seem ill-prepared to evaluate the answers.