The Survey Geek

The latest online panel dust up

Gregg Peterson's post earlier this week on this blog about the Panel of Panelists at the CASRO Online Conference created quite a stir. I saw an unusual number of pageviews, there was a fair amount of retweeting of the link and other industry commentators worked a similar theme. It came on the heels of Ron Sellers's More Dirty Little Secrets of Online Panels which also created quite a stir.

What I find surprising about all of this is, well, that people are surprised. For anyone paying half attention to the quality issues of online panels the core of what Gregg reported ought to be old news. I thought that we had already worked our way through the first four stages of the Kübler-Ross model and arrived at Acceptance. Recognizing that some people are doing an awfully large number of surveys and doing it for money is the least of it. This is convenience sampling run amok.

We've seen this movie before. A former colleague once spent much of his time on the road doing focus groups with IT managers. Time after time in every city it was always the same faces. He liked to call them "the Band of Brothers." He claimed to be on a first name basis with many of them. But the widely-acknowledged professional focus group participant did not completely undermine the usefulness of that methodology.

I've long been very critical of online and quick to point out its numerous shortcomings but I also think that we are better at it today than we have ever been. To the industry's credit, we finally have come to face its flaws and are evolving a set of standards and practices to deal with them.  We have a pretty good idea of how to do a better job of managing panels and are extending that to other non-panel sample sources. We have finally admitted that our sampling strategies need to evolve beyond age, gender and regional quotas. There are some very smart people helping us think our way through this. Granted, these developments are not yet industry wide. There are still way too many practitioners making outrageous claims, too many panel companies that have not embraced the kind of transparency that's needed for buyers to make sound judgments about quality and too many clients buying by the pound. Too much DADT.

We all know the old cliché about admitting you have a problem being the first step. I thought we were there.

The end of “don’t ask, don’t tell” in online survey research?

My colleague, Gregg Peterson, attended last week's CASRO Online Research Conference and has sent me this post.

The era of "don't ask, don't tell" in the world of commercial market research may well have ended last week in Las Vegas at the annual CASRO Online Research Conference.    The purveyors of online surveyors came face-to-face with the survey taking monsters they created.  Back in our rooms, if we looked up at the big mirrored walls of our swanky, conference hotel suites, we noticed the real culprits staring straight back at us.

Hands down, the talk of last week's conference was "the panel of panelists".  These were eight people recruited at the back of a large quantitative study of web survey takers who were there to help us understand their world and to tell us what they liked and disliked about the surveys we served up to them.  They were selected from among a few hundred qualified Las Vegas respondents (the Las Vegas respondents being a sub-quota of the large national study) based initially on the quality and thoughtfulness of their open ended responses in the quantitative survey, and, at a second stage, in a follow-up interview with the panel's eventual moderator, to see how well they could articulate their respective panel taking experiences.  (Let's hope that the recruiter was unconcerned about demographic diversity, because 7 of the 8 were not employed full time, none were below the age of 30, and all were Caucasians.)

And articulate they did - all of the scary sins of our industry.  Here's the least surprising of what we learned:  Each of them were members of multiple panels – the least ambitious of them was a member of a mere four panels while the majority of them seemed to be a members of 8-10 panels. When asked who had received multiple invitations to the same survey, all hands shot up immediately. A few happily admitted to taking the same survey multiple times.  Some seemed to be aware that it might not be completely kosher to do so, while one claimed proudly to have responded to all 10 invitations.  ("It's not up to me to police you guys.")  One very articulate elderly woman suggested that panel companies should allow them to indicate when they will be on vacation, because it was "so hard to come home to 600 or 800 survey invitations."  All were very clear about the importance of incentives and one panelist admitted to taking surveys and participating in focus groups simply to pay off the mortgage on his vacation home.   And he was less than happy about the new speeder detection tools sometimes imbedded in our surveys.    "It's easy to go fast when you've seen the same thing over and over again." 

As always, there are caveats. This was, after all, a qualitative study – not even pretending to be representative like most of the surveys we report on. That being said, there was some good news here.  These folks take their survey taking responsibilities seriously.  They did surveys at least in part because they really like giving their opinions.  They seemed sincere in their pledge of honesty and good intentions about providing truthful answers to all questions including screeners.  They only wish we were better at our jobs.  The spoke of broken links, screeners that take forever, highly repetitive surveys, progress bars that clearly lie, 50 minute surveys with too small an incentive, incentives that never show up, questions without a "Don't Know" category, missing response categories in our employment questions (retired people do not want to mark themselves as unemployed) and the dread of being confronted with dense text or a "a page full of tiny little boxes". They don't think their detergent has a personality and they were certain that there aren't 60 different ways to describe a soft drink.  A few really preferred surveys that gave them a chance to provide detailed open-end responses and many articulated the pleasure of taking a visually interesting and well-constructed survey.

What was perhaps most surprising was the reaction of researchers following this session. I heard a senior sales executive of a very large panel company express shock and amazement about the volume of surveys being taken and the professionalism of the respondents on the panel.  Everyone was abuzz.  Perhaps it was just the relief of getting it out in the open once and for all.  We could suddenly talk more freely about what we feared – the life blood of our industry may be a small army of highly determined professional survey takers. 

And guess what other topic was prominently featured on this year's agenda? Routers and maximizing panel "efficiency".  In other words, let's figure out if we can get these poor souls to do a few more poorly constructed surveys. We have work to do.

The bottom line on online sample routers

So what's the bottom line? I don't think we need to fear routing. Routing is essentially about automating and systematizing what panel companies have been doing for a decade. While some vendors are working on ways to claim greater representivity by the questions they ask in their screeners that's really a separate issue from routing. Likewise, sample quality solutions such as Imperium and TrueSample are distinct from routing.

So routing does not solve, nor does it pretend to solve, the underlying problems of doing research with volunteers, whether on panels or through other online sourcing methods. It probably does not make things worse and may create some significant improvements by bringing more discipline to the online sampling process and making it possible to broaden the sources we can use. What is certain is that the future is one with more routing, not less. My sense is that currently around 40% (give or take 10%) of US online sample is being routed. But I also expect that to continue to increase and eventually dominate.

Of course, the real bottom line is what effects if any we see in survey results using routed sample. It's not clear to me that that research has been done and I'm glad to see the ARF taking this on as part of their larger online sample quality initiative. Clients have made it very clear that the one thing they absolutely must have is reliability, replicability, consistency—whatever you choose to call it. They can't get one result this week and a different result with the same survey two weeks later. They can live without representivity or classic validity, but they must have reliability. I hope the ARF initiative helps us understand the role routing might play in meeting that need.

The problem of router bias

Everyone worries about router bias and it's not clear that anyone has figured out how to deal with it. At its core it seems to come down to the priority given to one survey over others and how that impacts the samples delivered to all of those other surveys. Before panels we drew samples that were independent of one another. But online samples create a dependency because sample drawing for a group of surveys is centralized, drawing on a fixed pool of respondents, in the case at hand, those entering the router. But, of course, the needs of one survey may conflict with those of another. I think of routers are chaos systems in action and each time a new survey is introduced you get a classic butterfly effect.

Here is a classic example. Let's say we have a general survey of car buyers as well as a survey of Hummer buyers both active in the router at the same time. Because of the low incidence of Hummer buyers all of them might go to that survey and the general survey of car buyers will have no Hummer buyers in it. (And if you happen to believe, as I do, that Hummer buyers have a unique perspective on the world that perspective, too, may be under represented in other surveys that have nothing to do with cars.) In other words, meeting the needs of low incidence surveys in the router could well bias the samples for the other surveys active at the same time. It gets really complex when you realize that there are hundreds if not thousands of surveys, all being serviced simultaneously by the same router and from the same pool of respondents.

Some argue that random assignment makes this problem go away but it's not clear if that's the case or how widely it's practiced. The fallback seems to be that this problem is inherent to panels, has always existed and has been managed inconsistently even within a single panel company. But everyone also agrees that it's a problem requiring a good deal more thought and empirical research before we understand it well enough to deal with it effectively.

Respondents in routers

The heart of routing is screening respondents. Conceptually, routing involves amassing all of the screening questions of waiting surveys (of which there typically are hundreds) and packaging them in a way that they can be efficiently administered while minimizing the chances that a respondent gets routed to a survey and then fails to qualify.  Some companies also use whatever information they already have on an individual to shorten the screening time. This might include profiling information on their panel members or, for non-panel members, information stored from a previous visit to the router by that individual.

One obvious concern is the fate of potential respondents in this process. Some people may take a long time to qualify and sometimes a respondent can pass the screener but still don't qualify in the survey proper. It's not uncommon to offer screen-outs, and occasionally completes, the opportunity go back into the router to try to qualify for another survey. Some companies place limits on how long they will let a person be in the router. Others limit the number of survey opportunities offered while some take the attitude that it's the Internet and people will do what people will do. Unlike real black holes, people can and do escape.

There are some major differences in the way all of this is managed. Going into the ARF meeting I thought I understood that there are three main types of router designs:

1. Serial routers in which screening questions for waiting surveys are asked one after the other until a respondent qualifies. These routers generally are smart enough to remember questions in common across the waiting surveys so that a specific question only gets asked once of each respondent.
2. Parallel routers in which a randomized set of screening questions is asked and then a respondent is routed to a survey for which he or she qualifies. This assignment might be random or it might be based on an algorithm that considers the needs of all waiting surveys.
3. Hierarchical routers which start with a main router where a few basic questions are asked and then people are passed to one of several mini-routers that feed surveys with similar qualifiers, on similar topics, etc.

In practice it's nowhere near that clean. Most companies seem to use a mix of hierarchical and serial routing. And everyone has arguments about why their approach is best.

In my next post I'll describe the problem that everyone worries about: router bias.