The Survey Geek

That old seat at the table

Attend an MR conference and chances are good that sooner or later a case of consulting envy will break out.  Symptoms include repeated use of phrases such as “seat at the table” and “C-level access.”  This is not a new phenomenon; it’s been a pretty steady part of the MR inferiority complex for the almost 20 years that I’ve been in the industry. The syndrome manifested itself at last week’s ESOMAR Congress in a session titled, “Think BIG: Imagine.”  It started from the premise that MR is twice as old and yet only one-tenth the size of management consulting (poor us) and then wondered how we might create “exponential growth” to close the gaps in revenue and prestige.  The answer, it turns out, is behavioral science.

To be clear, I agree that behavioral science is a megatrend with the potential to transform our industry.  But will its smart application make us rich and at long last bestow upon us that mythical seat at the table?  I doubt it. 

Management consultants and market researchers work at fundamentally different levels.  The consultant’s playing field includes corporate strategy and the full range of company operations, the whole enchilada.  Market researchers primarily provide tactical advice on how to take products to market. The two value propositions are fundamentally different and they result in fundamentally different ways of interacting with clients.

I am tempted to say that we should just get over it, enjoy the work that we do and take pride in the value we deliver. Then again, as tiresome as it can be maybe our fundamental insecurity is a good thing.  Maybe the constant fear of our lunch being eaten by management consultants and predictive analytics keeps us on our toes, pushes us to innovate, and makes us better if not bigger.  Andrew Grove probably was right; only the paranoid survive. 

I think we check that box.

Big Data at 3D

The topic has shifted to Big Data (BD from here on out) and moving from general talk to some tangible applications.

The first speaker was Jeff Hunter who showed us some specific uses of BD at General Mills. Well, maybe not so much BD as creative use of a whole range of different data sources, each fit to a specific problem related to company growth. One of his key points is reminding us that we live in a very rich data environment, much of which has not been mined and, sometimes, is free or very low cost. One of his examples was use of social media and sentiment coding to evaluate a new product trial. It worked well but there are caveats: the sentiment coding method makes a difference and the approach works best with high involvement products that generate lots of buzz. In another example he described what we might think of as "desk research" to gather a number of data series that were essentially free in order to evaluate a possible acquisition in an Eastern European country. Good, practical stuff from major buyer.

David Krajicek was up next and talked about whether MR has been doing enough to take advantage of Big Data. He used terms like "datification" and "digital exhaust." (I hope these don't stick.) He showed us a nice chart depecting the difference between BD and MR: Census/Sample, Flow/Fixed Point, Atheoretical/Hypothesis-Based, Unstructured/Structured, etc. The opportunity, as he sees it, is putting the two together and translating BD into "smart data." His argument is the current MR argument in this realm, specifically, that the skills of the market researcher and our classic concerns –representativeness, correlations vs. causality, complexity, etc. are essential here. The issue here is that value of BD is too great to leave it to the mathematicians. He showed some specific examples that merge survey data with BD-like data to enrich data and provide insights not possible from each source along. The thing is, we have been doing this for years. The presentation was a really good summary of the current storyline about BD in MR. Whether it is right, remains to be seen.

Finally, we had Greg Mishkin and his client, Don Hodson from AT&T who talked about a research program that goes through a cycle of interviewing, big data integration, and qualitative work as part of continuous cycle of improvement. One neat thing about this is that it includes a survey to assess whether the original model built by merging survey data with behavioral data is right as well as assess the impact of any actions the client may take to affect attitudes and even behavior. One key advantage of the BD approach that they highlighted is solving the recall problem inherent in surveys. Most importantly, the presentation has showed how surveys and big data can be used to great effect with a company that has a huge data base of interactions with their customers.

ESOMAR tackles Big Data

On Friday I attended a small gathering of around 30 MR suppliers, clients, privacy experts, and Big Data practitioners in Boston. The goal of the event was to stimulate a conversation about the practice and impact of Big Data on MR firms and the industry, to understand where we are headed, and the problems we are going have to solve to make it all work. The event kicked off with a presentation by John Deighton from the Harvard Business School who gave an excellent overview of the issues – current and future. His themes were not terribly different from those of the piece in Thursday's NYT, but he offered a good deal more detail with specific emphasis on implications for MR and marketing. The attendees then broke up into groups to develop positions on a set of pre-specified questions that were reported back to the group and discussed. In the afternoon we used a similar format, first with a panel of MR practitioners doing Big Data projects and then three privacy experts. 

I found this to be a really stimulating day. I would characterize the overall sentiment in the room as the day wore on as not being terribly different from the viewpoint shared by Larry Freidman in this blog post. The basic theme is that we now have a wide variety of sources and tools and we can serve clients best by bringing them together to tell a larger story. Some refer to this as "data diversity," although the term means different things to different people.

The revelation for me was in John Deighton's description of how data scientists who mine and analyzed these huge datasets approach their task. They are pure empiricists who eschew much of what we hold dear (e.g. accuracy, representativeness, association versus causality, etc.), all the skills that are the foundation of our profession. To quote Chris Anderson (editor of Wired), "[Big Data] makes the scientific method obsolete." We no longer need hypotheses or models; Big Data is self-modeling. We don't need to understand context or culture. We just need mathematics and the ability to listen to what the data are telling us. It is what it is. If it works, then the properties of the data don't matter. To my ears this sounds like atheoretical bullshit, but to a data scientist it is a first principle.

The Big Data future may be more challenging than we think, although it's not likely to be upon us as quickly as some fear. These kinds of transitions always take longer than we think. Or so some of us hope.

New survey says everybody has a price

Like many people in this business I've been thinking a lot about data privacy these last few weeks. So when I see a headline from Research-Live come into my email saying, "Half of consumers willing to share their data, says survey," I wonder what's up because it doesn't quite gel with other data I'm seeing. On close examination it's only 45% and there are other hedges as you go down through the piece.  Most importantly, the right verb for the headline probably is "sell" rather than "share." It turns about to be not terribly earth-shattering despite noble attempts by the sponsor and spokesperson for the company that did the work to make it sound special.

The real issue for me is not the numbers; it's whether I should believe any of what this piece says. I would like to know at least something about how this research was done beyond the N and the countries where people were interviewed. Was it online? How was the sample drawn? Who provided it? How were the questions worded? Was their weighting? And so on. I spent a few minutes searching the web for more info, but all I got was more links to the same unhelpful press release.

I don't mean to be singling out the good folks at Research-Live. This stunning lack of transparency is now commonplace in virtually all media channels. Online has made it possible for pretty much anybody to do a survey, whether they know or care about what they are doing or not. The web is awash in press releases with exciting findings from surveys, often with zero detail to help the reader understand whether those findings have any real meaning or are just cherry-picked from a bullshit survey. All of this is one more reason why the public has such low regard for surveys and why late night comedians can create an immediate giggle in their audience by saying, "There's a new survey out today . . ."

As it turns out, there is another survey on the same topic and with a similar finding:

According to the survey, 57 percent of consumers are willing to share additional personal information, such as their location, top five Facebook friends' names and information about family members, in return for financial rewards or better service, while 54 percent would even allow this data to be passed on to a third party, under the right conditions.

No details on the methodology used for this survey either. And I'm not going to jump to any conclusions just because the survey was released on the same day the sponsor announced a new suite of online data management and analytic products.

But I wonder whether the survey asked about throwing grandma in to get a better price?

Big data comes to the Census Bureau

Two of the more interesting sessions at last week's AAPOR conference featured the US Census Bureau. The shared theme was the Bureau's initiative to reengineer its data collection process in an era of declining cooperation and ever-tightening budgets. The two underpinnings of their strategy are (1) a new data collection approach called adaptive design and (2) big data.

Adaptive design is an enhancement to an earlier strategy called responsive design that replaces the traditional strategy of pursuing the highest possible response rate until either the money or time runs out. Adaptive design essentially says that the quality of the estimates is a better indicator of overall data quality than the response rate. To simplify it for the blogosphere, responsive design says that it makes no sense to continue to pursue interviews with certain types of people (say, a specific demographic group) if getting those data is not going to improve the survey's estimates, or at least the most important estimates. Adaptive design takes things a couple of steps further by saying that I can make decisions about which lines in my sample to pursue by using what I already know about them. Some of that information might come from a close monitoring of the field effort on the survey I'm running, and some might come from other sources. That's were big data comes.

The Census Bureau executes over 100 different surveys of households and businesses every year. Throw in the Decennial Census, and they have tens of thousands of field representative visiting or calling millions of homes and businesses, and learning at least a little something about each of them, whether they complete an interview or not. Putting all of this together in a systematic way will make it possible to separate out the easier respondents to get from the really hard ones. Bringing in data from the administrative records of other government agencies can enrich the database even further, sharpening the Bureau's ability to further prioritize the data collection effort. (I'm one of those people who believe that much of the Decennial Census might be done from these administrative records, but that's another post.) In theory, the survey effort becomes more efficient, can be completed more quickly, and will cost less.

But the Bureau faces the same challenge all users of big data must face: potential limits due to privacy protections. In their case it may come down to their ability to use data collected for another purpose. But unlike many of those other users, the Bureau approaches these issues with the utmost seriousness. Confidentiality protection is an obsession. The bar is significantly higher than simply what is legal. And so, they have an aggressive survey program designed to measure public attitudes toward an approach like what I've just described.

The jury is still out on all of this, but here's hoping they can make this work.