« Boys and toys | Main | What skills does a market researcher need? »

November 17, 2011


Feed You can follow this conversation by subscribing to the comment feed for this post.

I understand your point Reg and agree we need validation and consistent corroboration of these techniques, but I've been thinking for a while now that maybe there is something else at work here that is more difficult to quantify related to influence or herd mentality. I can't put my finger on it, but where there is smoke there is fire... Perhaps it is just the law of large numbers at play, but undeniably in some circumstances it does work.

@lennyism -- I was more or less with you right down to that last paragraph. It just ain't so. Less than a handful of one-off studies and people want to quickly make the leap of faith that because it worked once in this domain it will always work in all domains. Last year someone showed that Twitter can predict which new TV shows will be hits. This year it didn't. The problem with this line of thinking is that there are no theoretical underpinnings. It just works! Where have we heard that before?

Of course I can't disagree with your points Reg, but I think the salient point is "Anyone willing to spend the time and the money ...". Government and social research organizations often have the budgets to conduct that type of research whereas commercial research is often restricted on both time and money.

Moreover, I believe that the shift to emotional measurement/behavioral analytics, ethnographic models, and social listening are clearly indicative of businesses simply not finding probability samples necessary for their business.

The bulk of commercial survey research conducted today is based on convenience samples and we use a variety of techniques to try to compensate for that.

I agree that the science of sampling is important and has it's place, as do many research methods that have served us well in the past such as CATI, but increasingly they seem to be irrelevant (or at least of marginal importance) for most commercial research.

Also, in an age when twitter and Facebook can be used to exactly duplicate results of probability sample polls, that does indicate that perhaps we're being too dismissive of convenience samples, doesn't it?

I could not agree more with you, Reg. Moreover, even if good probability samples are getting harder to obtain or are not achievable in social media or mobile research, it does not imply that we (researchers) should give up on trying to get the best sample possible for a study. For example, probability samples have never been possible for mall intercepts. This didn't mean that you would interview the first 100 patrons coming through the door on Monday morning with no sampling strategy.

It's always been important to understand the limitations of one's sample and to try to unveil the effect those might have on the results; much more important than reporting a margin of sampling error. This fundamental research principle has not changed even if the emergence of new technologies offers different ways of collecting data.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.


Post a comment

Your Information

(Name and email address are required. Email address will not be displayed with the comment.)

Twitter Updates

    follow me on Twitter