When Voters Lie
The wireless only story still on hold?

Have we lost touch with reality?

No, this is not a comment on John McCain's surprising performance in the presidential polls.  Rather, it's the title of a recent one-day conference put on by the Association for Survey Computing in London. While  I admire these guys for their creative approach to meetings in places like The Old Doctor Butler's Head they also manage to bring together good people to talk about current survey issues. 

In this particular instance they brought together eight presentations around the theme of respondent engagement.  For those of you who have not been paying attention this is the new buzzword to describe attempts to improve online panel data quality by creating better online surveys, although generally "better" in this context has come to mean using Flash or similar tools to liven up the presentation of the survey and give respondents cool gadgets like slider bars and sorting exercises where you actually move things around in place of those old boring check box surveys that are the norm. 

Now I admit that I have been skeptical from the beginning because it all has been posited on the simple belief of researchers that Flash is better.  There never wre any clear and quantified signals from respondents calling for us to move in this direction.  And the fact that the main evangelists all have use of Flash-enabled surveys at the heart of their business models may be part of why I have been suspicious.  More importantly, we have tested slide bars in our work with the methodologists at UM and seem to continually find that you lose some respondents because of technical problems, that answering with these devices takes longer than with, say, radio buttons, and the data you get from them is no different.  Most proponents of Flash find much the same when they do systematic tests, but they often counter that with respondent satsifaction data that notes how cool and fun it is.

I invite you to review the papers and come to your own judgment.  And then at the risk of biasing you I would add that in various conversations I have had around the industry the feeling and more is that Flash is eye candy and not really a solution to the respondent engagement problem.  Our real problem is that our surveys are too long, or too complicated, or just really boring and Flash is another attempt, yes, to put lipstick on a pig.

Comments