The Survey Geek

Pop science is the new science of MR

Sometimes I worry that MR has gone 'round the bend. I had just such a moment yesterday during Ray Poynter's latest version of his online NewMR Festival. It was a great event with lots of good speakers on a wide range of topics with just the right amount of fun mixed in. One of those fun spots belonged to survey gamification enthusiast Jon Puleston who invented The Research Transformation Award, actually a series of awards across a set of categories ranging from ideas to software. Jon selected a couple of dozen industry worthies (including me) as judges, put out an online judging form and announced the results at yesterday's Festival. He also has posted the results on his blog where you, too, can be a judge.

Most of the winners were not surprising but the stunner for me was in the category of books. The top three vote getters were:

Information is Beautiful by David McCandless

The Wisdom of the Crowds by James Surowiecki

How We Decide by Jonah Lehrer

What all three of these books have in common is that they are written by journalists. It seems that the task of transforming MR has fallen to journalism, a craft once described by someone, maybe it was H. L. Mencken, as specializing in the "glib and superficial." The McCandless book doesn't bother me so much because it's mostly about storytelling and journalists are really good at that. But the other two, while entertaining reads, are classics of the pop science genre: a simplistic and easily grasped point of view loosely based in real science and relentlessly hammered home with a string of cherry-picked anecdotes. It is precisely the glib and superficial nature of these books that make them so appealing, but the problem is that in some cases at least "real scientists" aren't so sure that these books have got it right (see, for example, Adam Kepecs' s review of Lehrer's book in Nature. )

The most dramatic example of the danger here is the recent dustup around a study by Martin Lindstrom, author of another MR favorite pop science book, buy-ology. Lindstrom's study purports to show that we genuinely love our iPhones. Not just like, but love. According to the study, fMRI's showed that the ring of a subject's iPhone activated the insular cortex of the brain where we register feelings of love. Neat, except Lindstrom's got his science wrong as over 40 real neuroscientists were quick to point out.

Like most things in life if you really want to master something you've got to put in the time. Or, as an old high school teacher of mine used to say, "You are what you read." MR, please take note.

Surveys not dead yet

Last week I chaired the ESOMAR 3D event in Miami. Like most ESOMAR events it was awash in new thinking, innovative methods and lots of discussion, both formal and informal, about where the industry is headed. It was not just a breath of fresh air; it was the winds of change blowing hard in your face. In my summary at the end of the conference I noted that amongst all of the exciting talk about the methodological disruptions in MR there were at least small signs that rumors of the death of the survey are exaggerated.  Or was I just seeing what I wanted to see? And so I was cheered to see Lenny Murphy's excellent interview with Seth Grimes, a genuine thought leader in the field of text analytics.

I've been involved with surveys in one capacity or another for over 30 years and one of the dreams has always been software that can analyze and make sense of unstructured text. And so there is a certain irony that a method (social media listening) that many claim will replace surveys both relies on and has given renewed emphasis to the problem of making sense of text. But as in many instances with what we are now calling "the NewMR" we're not there yet. Or to quote Seth, "The race is on, but we are a long way from the finish line."

As his last question Lenny asked the big one: paint a picture of the future of MR. Here I quote Seth's answer at length:

I heard a speaker say, earlier this year, that with a "culture of listening" there is "no need for surveys." . . . No, you need surveys. For customer-experience initiatives, for market research, you can't learn everything you need to know without systematically asking a set of directed questions to a known set of respondents. Text analytics, sentiment analysis: These technologies will help you do better surveys. . . We'll see even further linking of survey- and social-sourced insights with behavioral and psychographic profiles inferred from "big data" clickstream, location, service utilization, transactional, and other tracking data and mined from content. This triangulation — ensemble methods that coordinate and combine multiple models and approaches — is the way to go.

Well said.