Well said

I'm catching up with some reading on a flight to DC and have come across an interesting interview in the May issue of Research World. While interviewing Obama's pollster, Joel Benenson, Simon Chadwick asks, "Is it all about how you analyze the data?" Benenson's response is especially eloquent:

"Look, if you write a good survey, you have written a survey to produce data that's going to be impactful in terms of your analysis and in terms of the decisions that you're trying to make. You have some hypotheses when you draft the survey, and you develop the best way to test those hypotheses. I view the questionnaire as an art form. That's where you really lay down the architecture of what your data could look like, and if you've done it well you're going to have a very rich data set."

Just yesterday I turned down an opportunity to be interviewed by someone about DIY surveys. As I said at the time, "I don't have any data to show how bad they are; mostly I just emote about it." My argument against them generally comes down to this: well-trained researchers add value because they know how to write good questions and good questionnaires. Benenson has said it so much better.

Unfortunately, it's not just about DIY. The MR industry has become so fixated on the analytics on the back end that we seem to have lost sight of the importance of how we get the data to begin with. Simon's question reflects that bias; Benenson's answer sets us straight.

Comments

[Brandon Watts]

Excellent comments. There is quite a bit of angst in the MR industry over the proliferation of DIY. Much of the angst is justified, as DIY is being applied to reduce costs, but the cost of data that is poor in quality, misleading, etc., may lead to much greater costs.

DIY isn't the problem, its the misapplication of these tools by those inexperienced in questionnaire design (and subsequent analysis) that is truly the issue.