Let’s have a look at online sample routers
February 10, 2012
A couple of weeks back I moderated about a four hour discussion on online sample routers sponsored by the ARF's Foundations of Quality research initiative. The current focus of the initiative is development of a research agenda that touches on the key dimensions of online research. The role of sample routers and how they may affect data quality is one of their issues. The discussion I moderated had about 15 people in it with reps from the major US panels as well as a handful of full-service research companies with large panels. As is often the case when you get a bunch of research types in a room talking about what they do it was a very open discussion with lots of sharing of practices, although generally not down to the detail level of secret sauces. This is the first of a whopping four blog posts summarizing what I learned.
I think it's fair to say that routers make a lot of researchers nervous. We have become comfortable with the standard online panel paradigm, in part because it's not unlike working with traditional list samples or even buying RDD . But routing is different if not downright mysterious to most of us. We imagine these streams of potential respondents from all of these disparate sources flowing into this black hole on the Internet and then pre-screened samples delivered to waiting surveys at the other end. Nobody, except the people who build routers, seems sure exactly what goes on in there, including me.
Like many people I have confused blending with routing. But the objectives of routing are different from blending. Blending is basically sourcing and how to be smart about leveraging your panel, river, social networks and other people's panels to create more diversity in a sample or to fill low incidence quotas. Routing doesn't necessarily involve blending; some companies are only routing their own panels.
Most routing tries to do four things (in no particular order):
- Maximize the likelihood that anyone who wants to do a survey can. Everyone sees the problems in continuing to send willing respondents to surveys for which they don't qualify.
- Increase the chances of filling all the quotas for every survey and delivering on time to clients.
- Automate sample allocation and deployment based on a set of well thought out rules so that the process is less ad hoc than in the past when it was done manually, often by project managers and in inconsistent ways.
- Centralize decision making about how to optimize use of the available pool of respondents and provide metrics so that these decision makers know what's going on inside the router.
Some would argue that there are other objectives like increasing representivity or improving sample quality but these are extraneous to the primary things that routers are designed to do.
In my next post I'll describe a little about how routers try to do what they are meant to do.