ESOMAR tackles Big Data

On Friday I attended a small gathering of around 30 MR suppliers, clients, privacy experts, and Big Data practitioners in Boston. The goal of the event was to stimulate a conversation about the practice and impact of Big Data on MR firms and the industry, to understand where we are headed, and the problems we are going have to solve to make it all work. The event kicked off with a presentation by John Deighton from the Harvard Business School who gave an excellent overview of the issues – current and future. His themes were not terribly different from those of the piece in Thursday's NYT, but he offered a good deal more detail with specific emphasis on implications for MR and marketing. The attendees then broke up into groups to develop positions on a set of pre-specified questions that were reported back to the group and discussed. In the afternoon we used a similar format, first with a panel of MR practitioners doing Big Data projects and then three privacy experts. 

I found this to be a really stimulating day. I would characterize the overall sentiment in the room as the day wore on as not being terribly different from the viewpoint shared by Larry Freidman in this blog post. The basic theme is that we now have a wide variety of sources and tools and we can serve clients best by bringing them together to tell a larger story. Some refer to this as "data diversity," although the term means different things to different people.

The revelation for me was in John Deighton's description of how data scientists who mine and analyzed these huge datasets approach their task. They are pure empiricists who eschew much of what we hold dear (e.g. accuracy, representativeness, association versus causality, etc.), all the skills that are the foundation of our profession. To quote Chris Anderson (editor of Wired), "[Big Data] makes the scientific method obsolete." We no longer need hypotheses or models; Big Data is self-modeling. We don't need to understand context or culture. We just need mathematics and the ability to listen to what the data are telling us. It is what it is. If it works, then the properties of the data don't matter. To my ears this sounds like atheoretical bullshit, but to a data scientist it is a first principle.

The Big Data future may be more challenging than we think, although it's not likely to be upon us as quickly as some fear. These kinds of transitions always take longer than we think. Or so some of us hope.