Still More on Shirking
The News from FedCASIC

Separating the Good from the Bad

It seems like every week I hear about a new Web panel.  One recent favorite whose name makes you wonder:

Our Sampling folks work hard to stay on top of this and to sort out the contenders from the pretenders.CASRO has been threatening to develop some standards to help us sort out the good from the bad, but ESOMAR is working harder to get out in front of the issue.  To spare us from reading through their entire code they've come up with 25 questions to ask a Web panel vendor.  Here they are verbatim:

  1. Is it an actively managed panel (nurtured community) or just a database?
  2. "Truthfully" how large is it?
  3. What is the percentage of "active" members and how are they defined?
  4. Where are the respondents sourced from and how are they recruited?
  5. Have members clearly opted-in? If so, was this double opt-in?
  6. What exactly have they been asked to opt-in to?
  7. What do panel members get in return for participating?
  8. Is the panel used solely for market research?
  9. Is there a Privacy Policy in place? If so, what does it state?
  10. What research industry standards are complied with?
  11. Is the panel compliant with all regional, national and local laws with respect to privacy, data protection and children e.g. EU Safe Harbour, and COPPA in the US?
  12. What basic socio-demographic profile information, usership, interests data, etc. is kept on members?
  13. How often is it updated?
  14. In what other ways can users be profiled (e.g. source of data)?
  15. What is the (minimum and typical) turn-around time from initial request to first deployment of the emails to activate a study?
  16. What are likely response rates and how is response rate calculated?
  17. Are or can panel members who have recently participated in a survey on the same subject be excluded from a new sample?
  18. Is a response rate (over and above screening) guaranteed?
  19. How often are individual members contacted for market research or anything else in a given time period?
  20. How is the sample selection process for a particular survey undertaken?
  21. Can samples be deployed as batches/replicates, by time zones, geography, etc? If so, how is this controlled?
  22. Is the sample randomized before deployment
  23. Can the time of sample deployment be controlled and, if so, how?
  24. Can panel members be directed to specific sites for the survey questionnaire to be undertaken?
  25. What guarantees are there to guard against bad data i.e. respondent cheating or not concentrating/caring in their responses (e.g. click happy)?