“The Skills Breakthrough”
Primary Primacy

The Latest Thinking on Panel Quality

Way back in October I spent two days chairing the ESOMAR Panels Conference in Orlando. Eighteen papers in all under four broad headings: Data Quality, Panel Quality, Proprietary Panels, and New Developments.

There was a lot of good stuff and some not so good, but one thing that really struck me is the gradual shift in thinking about the three main types of "bad respondents" that I and others have been talking about over the about the last two years.

First, there are the professional respondents who belong to multiple panels and do lots of surveys. More and more studies seem to be concluding that multipanel membership is not as problematic as we might have previously thought. Sure, you can find studies that point to some major demographic and behavioral differences but the overall plurality of studies seem to be finding few major effects. There are fewer studies of conditioning effects from taking lots of surveys but that work will come. Most of us would agree that fewer surveys are better than lots of surveys, but as a practical matter panel companies are going to continue to survey their members to the max as long as the demand is strong. So we need to get a better handle on how this might be impacting our results.

Second, there are the fraudulents, that is, people who misrepresent their qualifications either when they join the panel or when they are answering the filter questions in a survey. The panel companies (at least the reputable ones) seem to have embraced this as a serious problem that they need to own, and they are now happy to tell you how they are validating respondents at the registration step. As researchers we need to continue to police this and not hesitate to turn these people in to the panel provider when we find them in surveys. They are arguably the biggest threat to online research.

Finally, there are the inattentive, that is, respondents who complete quickly, don't really take the survey tasks seriously. Panel companies increasingly point to researchers as the ones who need to own this problem. Long, uninteresting questionnaires presented in formats that invite behaviors like satisficing are increasingly pointed to as the problem. This in turn is leading to a whole new industry-wide emphasis on "engaging respondents." The thinking seems to be that we can overcome long and uninteresting if we just present more creatively. More on that later.