The Survey Geek: A Perfect Storm

A Perfect Storm

June 27, 2006

We may have a perfect storm brewing in the world of Web panels. I've grabbed onto this often-used metaphor to describe the collision of three trends in MR, each of which is in direct opposition to the others. They are:

Increasing client focus on panel quality. The MR consulting firm Cambiar has reported in its last two annual surveys of clients that the top concerns are panel quality and cooperation rates. We are seeing that among some of our clients as well.
More MR firms deciding to buy rather than build. Surveys of MR practitioners also by Cambiar report that MR firms are beginning to shy away from building and maintaining their own panels, instead choosing to rely more on third party suppliers such as Survey Sampling, Greenfield, and e-Rewards. The implication here is that the panel business is brisk, and that is indeed the case despite Greenfield's woeful stock price.
Deteriorating panel quality Panels are being over used and response rates are cratering. This is a trend that we've observed in our own work going back to 2001. Anecdotally we know that panel members often receive multiple survey invitations in the same day. Some panel members are doing lots and lots of surveys. Studies have shown that there are panel effects, meaning that response quality deteriorates the longer people stay on panels and the more surveys they do. Panel vendors do not deny that cooperation is a problem and so is keeping up with the demand for respondents, especially in tough to get demographic groups.

So what's the next phase? I wish I knew. On thing is clear and it's that the choice of a vendor is more important than ever. It also seems clear that we need to understand panel response quality much better than we do and probably develop much better indicators of panel quality and respondent satisficing behavior. Finally, don't expect the panel companies to solve this problem for us. All signs are that they are going to ride their panel assets to exhaustion, or at least all the way to the bank.

Posted at 06:12 AM in Web Panels | Permalink | Comments (0)

Comments

Increasing client focus on panel quality. The MR consulting firm Cambiar has reported in its last two annual surveys of clients that the top concerns are panel quality and cooperation rates. We are seeing that among some of our clients as well.
More MR firms deciding to buy rather than build. Surveys of MR practitioners also by Cambiar report that MR firms are beginning to shy away from building and maintaining their own panels, instead choosing to rely more on third party suppliers such as Survey Sampling, Greenfield, and e-Rewards. The implication here is that the panel business is brisk, and that is indeed the case despite Greenfield's woeful stock price.
Deteriorating panel quality Panels are being over used and response rates are cratering. This is a trend that we've observed in our own work going back to 2001. Anecdotally we know that panel members often receive multiple survey invitations in the same day. Some panel members are doing lots and lots of surveys. Studies have shown that there are panel effects, meaning that response quality deteriorates the longer people stay on panels and the more surveys they do. Panel vendors do not deny that cooperation is a problem and so is keeping up with the demand for respondents, especially in tough to get demographic groups.