The Survey Geek

If we knew what we were doing

At one point yesterday my wife showed me a wall plaque in a Signals catalogue with a quote from Einstein that read, “If we knew what we were doing it wouldn’t be called research.”  For some reason, it struck a chord.  And then this morning I was reading a review of the new Malcolm Gladwell book in the Times.   Now I confess that I pretty much loathe this whole genre of pop science (including other books like The Wisdom of Crowds and How We Decide) that has taken over the non-fiction side of the best seller lists all the while presenting itself as something we need to take really seriously.  And there seems to be no shortage of people in MR doing just that.  So when I read Steven Pinker’s review of  What the Dog Saw I literally jumped from my chair with glee.  He nails not just the Gladwell book but the entire genre when he describes the reasoning in Outliers as “cherry-picked anecdotes, post-hoc sophistry and false dichotomies.”  Sure, it’s entertaining stuff.  But it’s also lazy, shallow, and maybe even dishonest.  And then I thought about all that is being written and said about “the new MR” and I began to wonder whether we might be heading down this same road.  I hope not.

Einstein also said, “It’s hard.  That’s why we call it research.”

Clients say the darndest things

For the past couple of weeks I have been mulling Stan Sthanunathan's remark at the IIR conference in Las Vegas reported as "quality doesn't matter."  I've also been getting reactions from friends and colleagues around the industry.  They vary from "I'm glad we don't work for Coke" to "How should this impact our strategy" with a midpoint of "He's just trying to make a point." 

I'm not going to reprise his entire argument here. That's why there's a hyperlink in the first sentence.  But it's worth a read because it seems to me to raise an interesting question, one that I have asked here before.  Do we really understand what level of precision clients need for the business decisions they make? There is this tendency for researchers to believe that when a client commissions a piece of research the results of that research and that research only drives a business decision. And so it needs to be spot on in terms of measurement accuracy or you get a bad decision.   But when you talk to clients they often tell you that they look at a whole range of data points as part of their decision making process and that one study we did for them is only one piece of a much larger puzzle.

There is, of course, at least one other explanation.  That would be that clients expect us to deliver accurate results.  Those are table stakes.  We differentiate ourselves through interpretation, not measurement.  Either way, these are tough lessons for a survey geek. 

The third rail of Resarch

Jeffrey Henning did such a nice job of blogging the ESOMAR Online Research Conference that I recommended his blog in a previous post.  But I've just read his summary piece on communities and worry that he has missed a key point, one that I tried to stress in no uncertain terms in my closing remarks. 

In virtually every conversation of any length about social media in MR and especially about branded communities the subject of marketing rears its ugly head.  The discussion in Chicago was no different.  The key point that way too many people don't get is that our ability to function as an industry is tied to our maintaining a clear distinction between research and marketing.  It's a line in the sand we dare not cross.  The third rail we dare not touch.

It was not that long ago in the US that we were sweating bullets because of the fear that MR would be subject to the then-pending National Do Not Call legislation.  It was widely understood that inclusion in the legislation would kill telephone research in the US.  We eventually were exempted because we do not deliver commercial messages.  Raise the same issue with European researchers and they will share their continuing fear of being lumped in with Direct Marketing by the EU and thereby suffering all sorts of restrictions that will strangle research.

MR is being transformed by technology and part of that has meant a whole new group of practitioners entering our industry, many of whom have yet to understand the long-term values of our industry.  While we welcome them we also need to educate them.  It's not a matter of some quaint ethical principles.  It's a matter of survival.

Might Web researchers be developing a conscience?

Amongst all the talk in Montreux about corporate social responsibility was some interesting discussion about how Web survey researchers also might develop a little social responsibility of their own.

First, there is Research-Voice.com. This is the brainchild of Kees de Jong (now CEO at SSI) who for the last couple of years has been championing what he calls a "respondent preservation initiative." The concept is simple. Without respondents there can be no surveys and so it is in the best interests of the industry to recognize that they are not a commodity to be used, abused, and cast off when we are done with them. We need to put respondents front and center and treat them with respect in all of our interactions with them. I certainly have spent some time in this space ranting about poor survey design and its impact on both respondents and response quality. It's a serious problem and Research-Voice tries to draw attention to it as well as encourage research-on-research aimed at improving the respondent experience. Check it out.

Then there is the "Pause for a Cause" initiative launched by the CMO Council. The idea here is to create a panel in which incentives go not into the respondents' pockets but to charities of their choosing. The goal is to redirect at least some of the money spent on online research into charitable donations. Needless to say, this idea has received a less than warm welcome by at least some companies in the panel business. Researchers also are expressing some concern that the motivations to join this panel (you can check it out here) might result in a whole new set of biases in its samples. A colleague has suggest that the better approach might be to offer the infrastructure to channel incentives to charities to all panel companies rather than launching a whole new panel composed of people with a sense of social responsibility.

I am rooting for Research-Voice to succeed but share the worries about bias in Surveyforgood. I like my colleague's suggestion about repositioning it as a service rather than a separate panel.

ESOMAR Congress

I've just returned from the ESOMAR Congress in Montreux.  This is  a great event to meet up with colleagues and partners, current and hoped for, from around the world.  But in all honesty, the content is not always as great as the networking opportunities.  I'm not going to comment on the overall content this year because I had a number of long meetings that caused me to miss to too many presentations.  But based on what I did see and conversation with other attendees it seemed to me that there were three overarching themes.

First, It was hard not talk about the economy, how bad it's been and when its' going to get better. The sub themes here are the same ones you hear everywhere--2010 a transition year; slow growth even after that in the US and Europe; faster growth in China and India; win business through innovation and efficiency; uncertain impacts of government regulation, etc.  All in all there were confirmatory messages with nothing especially new and startling.

Second, corporate social responsibility (CSR) is in.  At least ESOMAR believes it is because it was the main announced theme of the Congress, "Leading the Way: Ethically, Responsibly, Creatively."   A number of speakers asserted that the anxieties produced by Great Recession and the increasing emphasis on accelerating climate change are causing consumers to look more carefully at how brands are behaving relative to these issues and to factor that into brand choice.  While there was widespread agreement about the central concept there was considerable debate about the magnitude of the impact, especially from industry to industry.

Finally, there was this almost constant undercurrent of concern about two related issues that seem central to the European perspective on MR. The first of the issues is how we deal with respondents in a respectful and socially acceptable way.  This is about privacy, confidentiality, and the special problems of research with children.  The second is the need to continually define what we do and how we do it in ways that distinguish research from marketing.  I used to wonder why ESOMAR spends so much time on something like developing a definition of market research and now I get it.  It's ultimately about fending off regulation of the sort aimed at direct marketing.  This is a huge concern in Europe and researchers recognize that if we don't behave in an ethical and responsible way the freedoms we need to do our work could be seriously infringed upon.