The Survey Geek

Twitter and electoral polling: Not ready for prime time

Last night I chaired a panel at PAPOR on the general subject of social media and the 2012 election. The panelists were Paul Hitlin from PEW and Mark Mellman of the Mellman Group. The topic quickly narrowed to Twitter and the 2012 election. Both panelists had done substantial tracking of Twitter over the course of the campaign and showed multiple comparisons of Twitter volume, topics, and sentiment to polling results.

There were two main conclusions. Not surprisingly, while there were points of agreement between Twitter and the polls it was often weak, and at other times the two sources moved in opposite directions. Most striking was a Pew comparison showing a Romney surge on Twitter in the last week that seemed to say the he would win. (Hitlin started his presentation by claiming that if Twitter had been correct Ron Paul would be president!)

The second conclusion was more interesting. While the presenters agreed that Twitter did a poor job of tracking the campaign there were numerous disagreements between them, most of which they expected were rooted in their having used different software. Hitlin used Crimson Hexagon while Mellman used the Twitter-approved Topsy. So the results of both projects clearly demonstrated the degree to which the choice of software matters and the unpreparedness of many researchers at this point to make informed choices. No one in the room, including me, was prepared to debate the choice of one approach over the other. ESOMAR has tried to provide some help with its 24 Questions, but while we might know the questions to ask most of us seem ill-prepared to evaluate the answers.

#Twittersurvey: You knew this was coming

It’s been a big week for Twitter.  First came the announcement of the Nielsen deal and this item that seems to say Coke is embracing Twitter surveys in a big way.  Expect a stampede to follow.

This was inevitable but the timing is curious.  Pew tells us that as of May 15% of US adult Internet user say they use Twitter and just about half of those do so every day.  And those folks are disproportionately Afro-American, under 15 and live in urban areas.

You could look at this as one more reason for MR to work the worry beads, or you could see it as an opportunity.  After all, we’re supposed to be good at sampling, right?  And the basic principles of sampling are really useful for looking at a dataset, understanding its biases, and explaining who the data represent, what it means for the client’s target market, and therefore what actions the client should take. 

 But alas, the last 15 years of online research  has demonstrated pretty clearly that we don’t understand sampling much at all.  If we did, we would have recognized panels for what they are and either labeled the work appropriately or developed the techniques to overcome its shortcomings.  The later issue has finally moved to the top of the agenda for some, but sadly not for all.

So here we go again, ready or not.  Twitter, Facebook, Google, Mobile, Big Data – we are going to have to deal with all of it.  Will we dig into all of it in a systematic way to figure out what’s really there and what it can tell us or we will just accept it all at face value?  I’d like to think that one way for us to morph as these new data streams go mainstream is by leveraging our experience in research design and especially in the basic principles of sampling, not so we become samplers (God forbid) but so that we can evaluate the validity of the data in front of us.

It's hard to feel encouraged.  And if one more person tells me that it’s going to be ok because of the law of large numbers I surely will scream.  

Using Facebook to predict the yield on 10-year Greek bonds

The graphic below is from the "Correlation or Causation" section in the current issue of Businessweek.  I expect there is an ESOMAR paper not far behind. 

Another data point on social media privacy

The buzz about privacy protections in social media research has fallen off some since August's Great Privacy Debate. That doesn't mean the issue is settled by any means, only that the professional and trade associations who have been driving the debate have taken the feedback they've gotten and to one degree or another are back at the drawing board.

For me the key issue is not traditional survey research ethics, Terms of Use or legislative privacy protections. It's really about the expectations of social media participants. The industry's future will be bleak if we rely on methods that rightly or wrongly appear to abuse our research subjects. In this connection most of the survey data I've seen say that people have expectations about privacy in social media that are at odds with those of at least some MR practitioners.

So I found this little piece in The Market Research Bulletin to be interesting. In it some folks from Maritz report on a survey of Twitter users suggesting that when someone complains about a brand on Twitter the vast majority of those people hope that the brand follows up with them. I guess it's a cry for help. Only about a third report such follow-up and of that third 80% plus are pleased that they did. So in this instance there does not seem to be an expectation of privacy. They might not like it if their tweet is featured in a commercial, but they like the personal attention.

While I like the study I feel compelled to cry "foul" on a methodological point. The methodology description includes an estimate of sampling error which makes no sense with a nonprobability panel. While this is an all too common misuse of the statistic in MR it's a bit more objectionable here given that in the next paragraph Maritz boasts (justifiably) that they are ISO 20252 certified and the standard clearly specifies that standard error should only be calculated for probability samples.

Nonetheless, it's still an interesting little study.

"Twitter is outdirectedness cubed."

My wife is a borderline obsessive reader of the New York Times.  When she gets behind certain sections of it pile up around the house.  But in the end all of it gets read.  Some of it she clips and puts on my desk where it also piles up but in the end all of it, too, gets read.

Hence my late discovery of this interesting essay from July with the title, "I Tweet, Therefore I Am."  The essay refrences a forthcoming book by Sherry Turkle at MIT.  In it Turkle talks about how  posts to Facebook or Twitter gradually take on the character of a performance, an identify constructed for consumption by other people.  Not really your true self.  She reminds us of how the sociological masterwork The Lonely Crowd described the transformation of Amercian culture from inner-directed to outer-directed and how social media has sped that up.   Hence the title of this post, a quote from Turkle.

As any pioneer on the bleeding edge of the NewMR will tell you, we have embraced the basic tenant of behavioral economics that says people don't always do what they say they will do or can even explain after the fact why they did what they did.  This makes working with survey data a bit challenging, to say the least.  But by the same token, if people we listen to in social media are not true selfs but selfs constructed for our consumption, it raises some pretty significant challenges for working with social media information.

Or, to put it in terms a survey geek can understand we might say: Twitter is social desireaiblity on steroids.