The end of “don’t ask, don’t tell” in online survey research?
MRS Annual Conference

The latest online panel dust up

Gregg Peterson's post earlier this week on this blog about the Panel of Panelists at the CASRO Online Conference created quite a stir. I saw an unusual number of pageviews, there was a fair amount of retweeting of the link and other industry commentators worked a similar theme. It came on the heels of Ron Sellers's More Dirty Little Secrets of Online Panels which also created quite a stir.

What I find surprising about all of this is, well, that people are surprised. For anyone paying half attention to the quality issues of online panels the core of what Gregg reported ought to be old news. I thought that we had already worked our way through the first four stages of the Kübler-Ross model and arrived at Acceptance. Recognizing that some people are doing an awfully large number of surveys and doing it for money is the least of it. This is convenience sampling run amok.

We've seen this movie before. A former colleague once spent much of his time on the road doing focus groups with IT managers. Time after time in every city it was always the same faces. He liked to call them "the Band of Brothers." He claimed to be on a first name basis with many of them. But the widely-acknowledged professional focus group participant did not completely undermine the usefulness of that methodology.

I've long been very critical of online and quick to point out its numerous shortcomings but I also think that we are better at it today than we have ever been. To the industry's credit, we finally have come to face its flaws and are evolving a set of standards and practices to deal with them.  We have a pretty good idea of how to do a better job of managing panels and are extending that to other non-panel sample sources. RecalculatingWe have finally admitted that our sampling strategies need to evolve beyond age, gender and regional quotas. There are some very smart people helping us think our way through this. Granted, these developments are not yet industry wide. There are still way too many practitioners making outrageous claims, too many panel companies that have not embraced the kind of transparency that's needed for buyers to make sound judgments about quality and too many clients buying by the pound. Too much DADT.

We all know the old cliché about admitting you have a problem being the first step. I thought we were there.