Does it really matter how we present scales in Web surveys?
From the frying pan to the fire

The truth spoken here

Two interesting articles in the current issue of the trade rag, Research Business Report, both saying some things you don't hear said often enough.  First comes Bernie Malinof, founder of the Canadian consultancy Element54.  He sums up the current respondent engagement frenzy like this, "A 50-foot rogue wave is rapidly approaching with an explosion of interactive, rich-media questionnaires using formats that may or may not produce better data.  But I guarantee you they will produce different data. "

Readers of this blog will know immediately that I agree.  The experiments with this stuff that are routinely presented at industry conferences almost always focus on the fact that respondents like shinier online questionnaires, but few focus on whether or not the data we get is any better.  In my cynical moments I have been heard to say, "A lot of this Flash-enabled stuff is about people advancing their business models and not about producing better data."

Then here comes Kees de Jong, new CEO of SSI talking about, among other things, how we got ourselves into the panel data quality crisis. He contrasts the generally cautious approach of Europeans to online research with the full-steam ahead approach here in the U.S.  To wit, "But while that went on in Europe, Harris Interactive's George Terhanian and his peers convinced the U.S. market of the feasibility of online research.  Everyone began doing it, some without asking enough questions, and things overall got a little out of hand . . .U.S. culture is about business building and everyone wanting their piece of it."

And there you have the quandary.   How do you balance the need to innovate with the need to do good research?  How do you know when the legitimate desire to gain a competitive advantage in the marketplace crosses the line into snake oil? 

Comments