Online Sampling Again

Last week two posts on the GreenBook Blog, one by Scott Weinberg and a response by Ron Sellers, bemoaned the quality of online research and especially its sampling. And who can blame them? All of us, including me, have been known to go a little Howard Beale on this issue from time to time. We all know the familiar villains—evil suppliers, dumb buyers, margin-obsessed managers, tight-fisted clients, and so on. I was reminded of a quote from an ancient text (circa 1958) that a friend sent me a few months back:

Samples are like medicine. They can be harmful when they are taken carelessly or without adequate knowledge of their effects. We may use their results with confidence if the applications are made with due restraint. It is foolish to avoid or discard them because someone else has misused them and suffered the predictable consequences of his folly. Every good sample should have a proper label with instructions about its use.

Every trained researcher has an idealized notion of what constitutes a good quality sample. Every experienced researcher understands that the real world imposes constraints, that any one who says you can have it all—fast, cheap, and high quality—is selling snake oil. So we make tradeoffs and that’s ok, as long as we have “adequate knowledge of their effects.”

It helps to be an informed buyer, and ESOMAR has for about ten years now offered some version of their 28 Questions to Help Buyers of Online Samples to help. It’s an excellent resource, often overlooked. More recently, ESOMAR has teamed up with the Global Research Business Network to develop the soon-to-be-released ESOMAR/GRBN Guideline on Online Sample Quality (here I disclose that I was part of the project team). In the meantime, there is an early draft here. Its most prominent feature, IMO, is its insistence on transparency so that sample buyers are at least informed about exactly what they are getting and in as much detail as they can stand.

Granted, that’s not the same as “a proper label with instructions about its use.” That still is a job left to the researcher. If you can’t or won’t do that, well, shame on you. Getting depressed about it is not an option. Or, as another of the ancients has said, “You're either part of the problem or part of the solution.”