Previous month:
January 2012
Next month:
March 2012

Posts from February 2012

MR is not doomed, at least not yet

It’s Friday, an often slow day to start, and like many other people here in southeastern Michigan where I live I’m holed up at home with a  backpack full of work I brought home last night in anticipation of the crippling snow storm that was widely predicted but never came.  Catching up on my reading seemed like a good place to start and I was especially intrigued by the current issue or Research World.  The issue is titled, “Irrevocable Change: The digital revolution is transforming the market place.”  That title is a little hyped up for what follows because the issue itself is mostly a welcome relief from the apocalyptic pronouncements that have become the standard fare of any discussion of how MR should confront the economic, social and cultural changes flowing from global digitalization.  In his editorial Simon Chadwick sounds downright confident when he writes, “We have the talent, ability, methodologies and technologies to be able to manage the data deluge, and to provide the frameworks with which to analyze it and derive the insights which it is capable of providing.”  The articles in the issue taken as a whole seem mostly to argue for accommodation, augmentation and creative evolution rather than replacement, destruction and revolution.  We are indeed at an inflection point, as Simon describes, but that doesn’t mean we can no longer deliver value. It only means that we have to be smart and thoughtful about how the frameworks we choose genuinely help clients makes sense of the data deluge, because that, after all, is the point of it all.

As luck would have it, I subsequently came across this piece In Research by Tom Ewing that seems relevant to the process by which we choose those “frameworks” that Simon mentions.  Tom points out that with the help of the behavioral sciences we more or less have come to accept the idea that consumers don’t make completely rational purchase decisions, but that they are influenced by all sorts of things that we are only beginning to understand.    He invites us to view the decisions that we make as researchers about methodologies, for example, through the same lens we view the choices consumers make when we study them.  And he points out that when you do that you find that many of these decisions are, well, irrational in that they are less about how best to deepen a client’s understanding of a business problem than about ourselves—the methodologies we love, the need to appear hip and innovative.   

Living in a world with so much data and so many cool technologies is both a curse and a blessing.  As John Gambles says in the same issue of Research, “The industry has lost its way and allowed process to overtake purpose. And the value of research is not in the process but in its purpose.”  We probably should spend less time worrying about being gobbled up big new well-funded competitors who don’t know what we know about consumers and our clients’ businesses and more time remembering why this industry exists in the first place.

The bottom line on online sample routers

So what's the bottom line? I don't think we need to fear routing. Routing is essentially about automating and systematizing what panel companies have been doing for a decade. While some vendors are working on ways to claim greater representivity by the questions they ask in their screeners that's really a separate issue from routing. Likewise, sample quality solutions such as Imperium and TrueSample are distinct from routing.

So routing does not solve, nor does it pretend to solve, the underlying problems of doing research with volunteers, whether on panels or through other online sourcing methods. It probably does not make things worse and may create some significant improvements by bringing more discipline to the online sampling process and making it possible to broaden the sources we can use. What is certain is that the future is one with more routing, not less. My sense is that currently around 40% (give or take 10%) of US online sample is being routed. But I also expect that to continue to increase and eventually dominate.

Of course, the real bottom line is what effects if any we see in survey results using routed sample. It's not clear to me that that research has been done and I'm glad to see the ARF taking this on as part of their larger online sample quality initiative. Clients have made it very clear that the one thing they absolutely must have is reliability, replicability, consistency—whatever you choose to call it. They can't get one result this week and a different result with the same survey two weeks later. They can live without representivity or classic validity, but they must have reliability. I hope the ARF initiative helps us understand the role routing might play in meeting that need.

The problem of router bias

Everyone worries about router bias and it's not clear that anyone has figured out how to deal with it. At its core it seems to come down to the priority given to one survey over others and how that impacts the samples delivered to all of those other surveys. Before panels we drew samples that were independent of one another. But online samples create a dependency because sample drawing for a group of surveys is centralized, drawing on a fixed pool of respondents, in the case at hand, those entering the router. But, of course, the needs of one survey may conflict with those of another. I think of routers are chaos systems in action and each time a new survey is introduced you get a classic butterfly effect.

Here is a classic example. Let's say we have a general survey of car buyers as well as a survey of Hummer buyers both active in the router at the same time. Because of the low incidence of Hummer buyers all of them might go to that survey and the general survey of car buyers will have no Hummer buyers in it. (And if you happen to believe, as I do, that Hummer buyers have a unique perspective on the world that perspective, too, may be under represented in other surveys that have nothing to do with cars.) Hummerbuyer In other words, meeting the needs of low incidence surveys in the router could well bias the samples for the other surveys active at the same time. It gets really complex when you realize that there are hundreds if not thousands of surveys, all being serviced simultaneously by the same router and from the same pool of respondents.

Some argue that random assignment makes this problem go away but it's not clear if that's the case or how widely it's practiced. The fallback seems to be that this problem is inherent to panels, has always existed and has been managed inconsistently even within a single panel company. But everyone also agrees that it's a problem requiring a good deal more thought and empirical research before we understand it well enough to deal with it effectively.

Respondents in routers

The heart of routing is screening respondents. Conceptually, routing involves amassing all of the screening questions of waiting surveys (of which there typically are hundreds) and packaging them in a way that they can be efficiently administered while minimizing the chances that a respondent gets routed to a survey and then fails to qualify.  Some companies also use whatever information they already have on an individual to shorten the screening time. This might include profiling information on their panel members or, for non-panel members, information stored from a previous visit to the router by that individual.

One obMazevious concern is the fate of potential respondents in this process. Some people may take a long time to qualify and sometimes a respondent can pass the screener but still don't qualify in the survey proper. It's not uncommon to offer screen-outs, and occasionally completes, the opportunity go back into the router to try to qualify for another survey. Some companies place limits on how long they will let a person be in the router. Others limit the number of survey opportunities offered while some take the attitude that it's the Internet and people will do what people will do. Unlike real black holes, people can and do escape.

There are some major differences in the way all of this is managed. Going into the ARF meeting I thought I understood that there are three main types of router designs:

  1. Serial routers in which screening questions for waiting surveys are asked one after the other until a respondent qualifies. These routers generally are smart enough to remember questions in common across the waiting surveys so that a specific question only gets asked once of each respondent.
  2. Parallel routers in which a randomized set of screening questions is asked and then a respondent is routed to a survey for which he or she qualifies. This assignment might be random or it might be based on an algorithm that considers the needs of all waiting surveys.
  3. Hierarchical routers which start with a main router where a few basic questions are asked and then people are passed to one of several mini-routers that feed surveys with similar qualifiers, on similar topics, etc.

In practice it's nowhere near that clean. Most companies seem to use a mix of hierarchical and serial routing. And everyone has arguments about why their approach is best.

In my next post I'll describe the problem that everyone worries about: router bias.

Let’s have a look at online sample routers

A couple of weeks back I moderated about a four hour discussion on online sample routers sponsored by the ARF's Foundations of Quality research initiative. The current focus of the initiative is development of a research agenda that touches on the key dimensions of online research. The role of sample routers and how they may affect data quality is one of their issues. The discussion I moderated had about 15 people in it with reps from the major US panels as well as a handful of full-service research companies with large panels. As is often the case when you get a bunch of research types in a room talking about what they do it was a very open discussion with lots of sharing of practices, although generally not down to the detail level of secret sauces. This is the first of a whopping four blog posts summarizing what I learned.

I think it's fair to say that routers make a lot of researchers nervous. We have become comfortable with the standard online panel paradigm, in part because it's not unlike working with traditional list samples or even buying RDD . Lifes-maze But routing is different if not downright mysterious to most of us. We imagine these streams of potential respondents from all of these disparate sources flowing into this black hole on the Internet and then pre-screened samples delivered to waiting surveys at the other end. Nobody, except the people who build routers, seems sure exactly what goes on in there, including me.

Like many people I have confused blending with routing. But the objectives of routing are different from blending. Blending is basically sourcing and how to be smart about leveraging your panel, river, social networks and other people's panels to create more diversity in a sample or to fill low incidence quotas. Routing doesn't necessarily involve blending; some companies are only routing their own panels.

Most routing tries to do four things (in no particular order):

  1. Maximize the likelihood that anyone who wants to do a survey can. Everyone sees the problems in continuing to send willing respondents to surveys for which they don't qualify.
  2. Increase the chances of filling all the quotas for every survey and delivering on time to clients.
  3. Automate sample allocation and deployment based on a set of well thought out rules so that the process is less ad hoc than in the past when it was done manually, often by project managers and in inconsistent ways.
  4. Centralize decision making about how to optimize use of the available pool of respondents and provide metrics so that these decision makers know what's going on inside the router.

Some would argue that there are other objectives like increasing representivity or improving sample quality but these are extraneous to the primary things that routers are designed to do.

In my next post I'll describe a little about how routers try to do what they are meant to do.