The Survey Geek

ISO 20252 – Market, opinion and social research

In December we were audited for ISO compliance. After a week of staff interviews, project reviews and records inspections we were certified to the 20252 standard. Good for us!

Now I recognize that there are more than few people in the industry who think ISO certification is some kind of retro thing that does not reflect how much the industry is changing. More on that in a bit. But first let me say that when it's all said and done ISO 20252 comes down to four pretty basic things:

1. Making sure that people know how to do their jobs and can succeed in the assignments we give them.
2. Ensuring that everyone on a project team understands their individual roles, the roles of their colleagues and how to work as a team to produce great work for their client.
3. Being completely transparent with clients about what was done and how it was done so that they can make whatever business decisions they need to make with confidence.
4. Minimizing the mistakes that generate rework and disappoint clients while maximizing the chances of learning from those mistakes.

Now back to those folks who say that ISO is for the dinosaurs and out of place in the fast-moving world of contemporary market research. Or that all that process ISO insists on works against the innovation that is the supposed life blood of boutiques and startups. I think that's nonsense. A well-defined infrastructure of process and procedures need not be a barrier to change. In fact, it can make it easier. In Good to Great Jim Collins observes that in great companies people work within the framework of a highly developed system that fosters what he calls "a culture of discipline." When you have a culture of discipline you spend less time managing people and more time managing the system. And when change is called for you change the system, not the people. That's also what ISO brings you. I recommend it to you one and all.

Eric Salama on Radio NewMR

This morning I listened to an interview of Eric Salama, CEO of Kantar (and now TNS as well), on Radio NewMR. I know Eric Salama by name only and was interested to hear his take on the future of the industry.

Overall his message seemed to be that our industry should focus on its strengths rather than becoming distracted by worries about big players from other industries coming in and eating our lunch. There are now many sources of insight that go beyond the world of surveys and focus groups. MR might exploit some of them but he sees no mandate to exploit them all. Four things in particular stood out in my mind:

• Surveys are not dead yet. There are some things you can only learn by asking people questions.
• There sometimes is a disconnect between what we hear about the explosion of new methods and client spend. His specific example was social media whose current share of client spend he characterized as "unimpressive."
• MR firms generally lack the technology to compete with major players like IBM when it comes to Big Data. Crunching massive databases of unstructured data is not what we are good at. We are good at analysis and that's where we should focus.
• Qualitative is a major opportunity, at least for a big player like Kantar. He sees the qual side of the industry as still very fragmented and a place where there has been a good deal of innovation still to be leveraged.

The interviewers—Ray Poynter and Lenny Murphy—did their best to get him to bite on more apocalyptic futures for MR but he would have none of it. Still, it was an interesting half hour. But don't take my word for it. You can hear a recording of this morning's show along with previous shows here.

Amsterdam Syndrome

Back in September I asked my friend and colleague, Theo Downes-LeGuin, to write a blog post that summarized his reaction to the ESOMAR Congress.  He wanted to take some time to let it all sink in and has now sent me something.  Here it is.

After 2 days at the ESOMAR Congress in September, I must conclude that I am washed up, a change-averse has-been. Yes, there is evidence to the contrary. Though trained in classical survey methods I've spent my career with one foot in qualitative research. I adopted disk-by-mail methods early, before the internet (quaint!). I was an early proponent of online qual despite client resistance. I like to think that I contributed to many of now-standard approaches for maintaining data quality in web panel surveys. And of late I've been working on how we make online experiences more engaging for online insights providers (née respondents).

But over 48 hours in Amsterdam, I was told by enough speakers and delegates that to even think about, let alone discuss, the themes that have occupied me for the past 20 years is to consign myself to the Pleistocene Era. And if you hear something enough, you start to believe it's true. Call it Amsterdam Syndrome.

After a few weeks at home, the syndrome is wearing off. I don't know about you, but I get a bit churlish when lots of people start telling me that we're doomed if we don't do X (where X at the moment could mean abandon surveys, replace representivity with relevance, or gamify everything). The end is nigh, as the wild-eyed man with the sandwich board has been telling us for 50 years, and one of these days he'll be correct. But at the moment, more likely we're simply replacing one orthodoxy with another.

Our new orthodoxy hasn't quite taken shape yet, beyond the assertion that Market Research Must Change In Order to Survive – a statement so obvious and enduring as to be hardly worth repeating, at least no more worth repeating now than it was 20 years ago (yes, there was turmoil and risk of irrelevance then as well). Any researcher worth his or her salt is constantly exploring poking at sacred cows and improving the status quo. Sometimes we have to explore a little harder due to changing environmental factors, and some changes are bigger than others.

But a new orthodoxy will surely will take shape, and it will take its cues from business, not from the sciences. Most of the time business serves MR well as a model, both because our raison d'être is to serve other businesses and because we are ourselves businesses. But the model falls apart a bit when we are trying to judge how we should change in our own practices. Businesses are by definition opportunistic; they share very clear measures of success (growth and profitability); and in the current environment they are very responsive to the immediate environment.

Scientific principles, on the other hand, are really handy when it comes to change management. God knows, science is subject to its own orthodoxies. But in between those conflicts and upheavals, scientific principles and process provide a consensus basis on which design experiments and evaluate outcomes, all in order to decide which changes are worthwhile and which are rubbish. Which I believe is what we are paid to do for our clients…so why wouldn't we do it for ourselves?

Pop science is the new science of MR

Sometimes I worry that MR has gone 'round the bend. I had just such a moment yesterday during Ray Poynter's latest version of his online NewMR Festival. It was a great event with lots of good speakers on a wide range of topics with just the right amount of fun mixed in. One of those fun spots belonged to survey gamification enthusiast Jon Puleston who invented The Research Transformation Award, actually a series of awards across a set of categories ranging from ideas to software. Jon selected a couple of dozen industry worthies (including me) as judges, put out an online judging form and announced the results at yesterday's Festival. He also has posted the results on his blog where you, too, can be a judge.

Most of the winners were not surprising but the stunner for me was in the category of books. The top three vote getters were:

Information is Beautiful by David McCandless

The Wisdom of the Crowds by James Surowiecki

How We Decide by Jonah Lehrer

What all three of these books have in common is that they are written by journalists. It seems that the task of transforming MR has fallen to journalism, a craft once described by someone, maybe it was H. L. Mencken, as specializing in the "glib and superficial." The McCandless book doesn't bother me so much because it's mostly about storytelling and journalists are really good at that. But the other two, while entertaining reads, are classics of the pop science genre: a simplistic and easily grasped point of view loosely based in real science and relentlessly hammered home with a string of cherry-picked anecdotes. It is precisely the glib and superficial nature of these books that make them so appealing, but the problem is that in some cases at least "real scientists" aren't so sure that these books have got it right (see, for example, Adam Kepecs' s review of Lehrer's book in Nature. )

The most dramatic example of the danger here is the recent dustup around a study by Martin Lindstrom, author of another MR favorite pop science book, buy-ology. Lindstrom's study purports to show that we genuinely love our iPhones. Not just like, but love. According to the study, fMRI's showed that the ring of a subject's iPhone activated the insular cortex of the brain where we register feelings of love. Neat, except Lindstrom's got his science wrong as over 40 real neuroscientists were quick to point out.

Like most things in life if you really want to master something you've got to put in the time. Or, as an old high school teacher of mine used to say, "You are what you read." MR, please take note.

Surveys not dead yet

Last week I chaired the ESOMAR 3D event in Miami. Like most ESOMAR events it was awash in new thinking, innovative methods and lots of discussion, both formal and informal, about where the industry is headed. It was not just a breath of fresh air; it was the winds of change blowing hard in your face. In my summary at the end of the conference I noted that amongst all of the exciting talk about the methodological disruptions in MR there were at least small signs that rumors of the death of the survey are exaggerated.  Or was I just seeing what I wanted to see? And so I was cheered to see Lenny Murphy's excellent interview with Seth Grimes, a genuine thought leader in the field of text analytics.

I've been involved with surveys in one capacity or another for over 30 years and one of the dreams has always been software that can analyze and make sense of unstructured text. And so there is a certain irony that a method (social media listening) that many claim will replace surveys both relies on and has given renewed emphasis to the problem of making sense of text. But as in many instances with what we are now calling "the NewMR" we're not there yet. Or to quote Seth, "The race is on, but we are a long way from the finish line."

As his last question Lenny asked the big one: paint a picture of the future of MR. Here I quote Seth's answer at length:

I heard a speaker say, earlier this year, that with a "culture of listening" there is "no need for surveys." . . . No, you need surveys. For customer-experience initiatives, for market research, you can't learn everything you need to know without systematically asking a set of directed questions to a known set of respondents. Text analytics, sentiment analysis: These technologies will help you do better surveys. . . We'll see even further linking of survey- and social-sourced insights with behavioral and psychographic profiles inferred from "big data" clickstream, location, service utilization, transactional, and other tracking data and mined from content. This triangulation — ensemble methods that coordinate and combine multiple models and approaches — is the way to go.

Well said.