Surveys not dead yet

Last week I chaired the ESOMAR 3D event in Miami. Like most ESOMAR events it was awash in new thinking, innovative methods and lots of discussion, both formal and informal, about where the industry is headed. It was not just a breath of fresh air; it was the winds of change blowing hard in your face. In my summary at the end of the conference I noted that amongst all of the exciting talk about the methodological disruptions in MR there were at least small signs that rumors of the death of the survey are exaggerated.  Or was I just seeing what I wanted to see? And so I was cheered to see Lenny Murphy's excellent interview with Seth Grimes, a genuine thought leader in the field of text analytics.

I've been involved with surveys in one capacity or another for over 30 years and one of the dreams has always been software that can analyze and make sense of unstructured text. And so there is a certain irony that a method (social media listening) that many claim will replace surveys both relies on and has given renewed emphasis to the problem of making sense of text. But as in many instances with what we are now calling "the NewMR" we're not there yet. Or to quote Seth, "The race is on, but we are a long way from the finish line."

As his last question Lenny asked the big one: paint a picture of the future of MR. Here I quote Seth's answer at length:

I heard a speaker say, earlier this year, that with a "culture of listening" there is "no need for surveys." . . . No, you need surveys. For customer-experience initiatives, for market research, you can't learn everything you need to know without systematically asking a set of directed questions to a known set of respondents. Text analytics, sentiment analysis: These technologies will help you do better surveys. . . We'll see even further linking of survey- and social-sourced insights with behavioral and psychographic profiles inferred from "big data" clickstream, location, service utilization, transactional, and other tracking data and mined from content. This triangulation — ensemble methods that coordinate and combine multiple models and approaches — is the way to go.

Well said.

Comments

[Michael_Deis]

I agree with your view and it is a mature and reasonable position to take. My fear is that we are losing the capability to even offer probability samples in the on-line and mobile space.

My work requires the capacity to project to populations. When looking around the world, I am unable to find an on-line probability sample. My hope was that as we began to see a move to mobile/cellular telephone surveys, we would see a return to fundamentals in the area of sampling but this does not seem to be the case. At the moment, I can still find face-to-face and RDD samples depending on the country.

With the advent of mobile technology and its ubiquity, especially in Asia, I hoped that we would see probability sampling reassert itself. This is the future of the telephone survey and our profession seems to be leaving behind the scientific basis upon which we can make projections without a debate on the issue.

I am hoping your blog might be a place where we can raise some awareness on the issue and pursue a reasoned discourse. The more we get our colleagues thinking about the issue the better off we will be.

[Reg]

@michael -- While have ranted a lot in this space about the failure of online to produce representative samples I think we also need to recognize that a probability sample is not always required. If you are testing a new snack food concept maybe a demographically diverse group of dedicated snack food eaters is enough. That is, as long as we don't tell the client the sample is something that it's not. That may not be the kind of work that I enjoy doing, but it's not necessarily evil. Truth in sampling is the key for me.

[Michael_Deis]

Wish I was as optimistic as you. Even if we still have surveys, how many actually attempt to employ probability sampling. No one to my knowledge, except Knowledge Networks, even pays lip service to proper sampling in the on-line space. And as the industry moves to mobile survey research, I see not one champion in the research community for a return to the fundamental practices that allowed us to project to populations and gave us a scientific leg on which to stand. How do opt-in, self-selected samples provide any basis for optimism that surveys are better than "listening" to social media? No one I ran into at the Esomar event is even thinking about the consequences of our industry’s move away from the bedrock principal of probability sampling for surveys.

[Leonard Murphy]

Thanks for the shout out Reg! It was great seeing you last week and agree; the winds of change were blowing like a hurricane out of Miami.