The Survey Geek

Why does it work?

Now there is a question that we ought to ask more often. At last week's MRMW Conference we heard about a whole lot of different methods—some mobile and some online—for which their evangelists made great claims. And those claims generally focused on the great insights they can deliver about some generic ill-defined group, typically "people" or "consumers." It's not always clear about which people or which consumers we're learning about. Does it mean everybody or just somebody? If somebody, exactly who?

So around the end of the second day after still another presentation describing an approach to online research that delivers incredibly accurate results for or an equally incredibly low price without all that messy sampling and weighting I asked the question: Why does it work? After a pause I got my answer: "Because we are very careful."

Say what you will about probability sampling but you have to admit that is has a theory underneath it, some scientific underpinnings and lots of empirical research to make a strong argument for why it works. Violate its key assumptions through incomplete coverage or high nonresponse and it may not work so well. The New MR crowd has the arguments more or less down pat (although a fair number of them seem not to have gotten the memo about sampling mobile phones now being SOP).

But If we are doomed to non-probability sampling methods then let's at least hold them to the same standard. Let's ask why it works. Let's ask how we are to know that the methods used to collect the data and analyze it lead to a reasonably accurate measure of the attitudes and the behaviors of the people the research claims to represent.

Let's start having those conversations. "It just works" should not be enough. That crosses the line from science to faith, and we all know what Mark Twain said about faith.

Final thoughts on MRMW

In the final session of last’s week MRMW Conference chair Lenny Murphy asked me to share my three takeaways from the conference.  Here is more or less what I told him.

First, if you are in the data collection business find a new business.  It’s clear that the sheer amount of data available is going to keep expanding exponentially.  Their quality may not be great either as a basis for decision making or as measured by traditional MR standards, but if online and social media have taught us nothing else it’s that clients sooner or later will buy cheap data over good data every time.  The value for MR will be where it always should have been: filtering, analyzing and helping clients put it to work.

Second, it’s time to put some bounds around how we use those data.  At the top of this list is privacy.  The fastest way to slay the golden goose is to abuse the owners of all those data.   There is certain cluelessness on this issue in some of what I see presented at this and other conferences.  One thing that will not change is our reliance on people—the public—to make our methods work.

Third, clients need to become smarter consumers.  To paraphrase Lyndon Johnson, they need to be able to tell the difference between chicken shit and chicken salad.  There is a lot of hand waving about what some of these new methods do and why what they eventually tell us has value, but precious little proof that that’s the case.  So clients will need to dig deeper and demand more. 

On the ride home a fourth occurred to me.  Could it be that in the era of big data we all will become qualies?  Think about it.

Storytelling comes to the conference podium

The first two presentations at the conference this morning have helped me to understand what it is that I'm finding so disquieting about this conference. What we are hearing is not the usual kind of research conference presentation. What we are hearing are stories. No data. No experiments. No hypothesis testing. Just stories. Oral blog posts from the podium with some neat visuals by people who are smart and articulate. 

OK, so I probably am exaggerating some but it makes me wonder whether this is where the current emphasis on "storytelling" is taking us as an industry. And, of course, is that a bad thing? Are we really connecting the dots or just making unconnected leaps of logic to get to a view point? In my previous post I worried that we are too focused on "just the facts, ma'am" as a deliverable. We probably should be just as worried about wonderfully-sounding insights that are disconnected from those facts. 

 

Kids in a candy store

I'm at the Market Research in the Mobile World Conference in Cincinnati where the constant stream of cool MR mobile applications is off the charts. You can't help but be impressed with the creativity being brought to the task of using mobile to give us a whole new perspective on consumer behavior. And the people showing their wares here in Cincinnati all seem to be having a great time doing it. The excitement is palpable.

But the most interesting session of the day to my ear was the client panel to which our host and MR impresario Lenny Murphy posed the question: "What's the gap between what clients need and what suppliers are offering?" The answers all seem to boil down to our failure to help them understand how all of this cool new stuff and the data it produces can be put to work in their companies. There were lots of specifics—boring reports that need to be rewritten, failing to help internal stakeholders integrate the data into their existing databases, not being clear about how new methodologies do what they purport to do, not understanding their business and industry, etc. It's certainly not the first time we've heard this from clients. And I've seen zero evidence so far that mobile helps.

Clients have been complaining for years that researchers mostly just deliver facts, or what they claim to be facts, and making real business sense of those facts is not a deliverable. For all the talk about the importance of storytelling, it's just another way to present the facts. Ditto for most of the cool video and animated deliverables being generated out of mobile. Clients want us to connect the dots from those facts to their business and they keep telling us over and over that we're not getting it done. I guess we're just having too much fun with data collection.

Ying versus yang

NCHS just released the latest data on US wireless only households. The relentless march continues and as of December, 2011, a whopping 34% of US households have only a wireless telephone. To put it another way, you can only hope to reach about two-thirds of the US population when only calling landline telephones. Clear evidence, I'm sure, that telephone surveys are dead. 

Pew also  just released a report saying that 49% of US adults use their cell phones to go online. At this rate about two-thirds of US adults will be using their mobile to go online in 2014. More proof, I'm sure, that mobile's time has come.