Six of one . . .

We do a large amount of survey work with physicians and virtually all of it relies on online panels. Not that long ago we had a client who insisted on our using mail to recruit physicians to Web surveys, partially on the premise that it would produce a more representative sample. That client has since backed away for that insistence although the reason probably has more to do with turnaround and perhaps cost than sample quality. Now a colleague who works with that client has asked about the pros and cons of mail recruit to Web versus use of an online panel. It's a hard question.

There are about 850,000 licensed physicians in the US. A first rate online physician panel will generally have enrolled about 150,000 of them, or about 18% of the total. Putting the invitation to join in front of most of those 850,000 is doable and it probably is safe to assume that most companies specializing in physician sample have done just that. So in statistical terms, the response rate to the recruit stage of a survey using the panel is 18%. Response rates for the physician panels we work with are typically in the 25%-30% range. The overall response rate is the product of the recruitment response rate and the individual survey response rate or around 5%. Back when we were mailing to samples of physicians our typical response rate was also 5%.

So which is the better sample? To answer that question we need to know a lot more than we now know. Both samples have huge amounts of nonresponse (95%) and unless we know a good deal about the factors that drive that nonresponse and their possible interaction with the specific study topic it's impossible to say whether one sample is better (i.e., more representative than the other. The time honored way to answer that question is to select a small sample of nonresponders and work it aggressively to try to uncover the dynamics of nonresponse. That's a bit more complicated in the case of the online panel because there are two stages of nonresponse—original recruit to the panel and the individual survey—whereas the mail method has just the one stage of the individual survey. Just as with consumer panels, we understand precious little about the attitudinal and behavioral characteristics that drive some people to join an online panel while the vast majority of others do not.

I can imagine a methodological study that would shed some light on all of this but relatively little of this kind of work is done because working with physicians is so expensive, honoraria easily $100 or more. That's a real shame because we should know more about these dynamics.

So the Pharma industry marches on relying almost exclusively on online panelists about whom we know relatively little. To their credit, these respondents are very practiced at doing very complex surveys (as are panel respondents generally) loaded with difficult exercises (such as discrete choice or marketing message review) that may take 45 minutes or more to complete, albeit for incentives that are 10 or more times what we would pay a consumer. And there is little evidence to argue that doing so will produce different results than what we would get from a classic probability sample from a high quality frame.