The Survey Geek

And now, for a little pointless rambling

I've never been one to spend a whole lot of time agonizing about the state of the research profession or the kinds of people we should have in it. But a few weeks back when I posted something on eye candy in Web surveys it attracted a comment from Chris Ryan that raised the more general issue of the usability of Web surveys, whether plain or fancy. The obvious implication of the comment (at least to me) is that researchers are busy designing these graphical presentations (plain and fancy) of survey questions and arguing about design when in most cases we really don't know a whole lot about usability. We are, in fact, mostly amateurs.

This reminded me of a conference talk I heard Pete Cape from SSI give earlier in the year in which he started by asking everyone in the room to stand up. Then he asked everyone who had wanted to be in MR since they were a kid to sit down. Nobody sat down. Next he asked about high school and then college and then grad school and when he finished most of the room was still standing. His point was that most of us did not aspire to careers in research and few of us were actually trained in school to do it, we just have sort of fallen into it. We are, in fact, mostly amateurs.

I freely admit that the "amateur" tag works on me as well. I have a Ph.D. in History, which is not exactly the ideal training ground for market researchers. But at least I had the good fortune to spend over a decade at a place (NORC) loaded with first class survey researchers and survey methodologists. I learned an enormous amount in that time and, as corny as it sounds, some of my best friends are survey methodologists. None of this made me a good survey methodologist, but at least I know good work when I see it and, frankly, the more I look around the industry the less really good survey work I see.

Then I read a short opinion piece in IJMR by Anthony Tasgal arguing that what MR needs is "more seers, fewer craftsman." Unlike me, he sees us an industry of skilled craftsmen (and craftswomen) churning out our work like the loyal and hardworking Boxer in Animal Farm. He makes the point that we are living in a time of dramatic change in marketing, communications, and consuming habits brought on principally by the Web and our industry must change with it. To effect that change, we need different kinds of people, fewer craftsmen and more creative types. Seers. To that end he tells us to send our "people out to art galleries, take them to the cinema, give them non-marketing books to read, and allow them to turn their wandering into wonder."

In the same talk I mentioned above Pete Cape said, "We have allowed our industry to be taken over by venture capitalists and technology geeks." Given that perspective, more seers would be a real improvement. But let's be careful. It seems to me that the MR industry has always rested on our ability to assemble information, evaluate evidence, and draw conclusions that help our clients make good business decisions. Mostly that has meant doing surveys. I am willing to accept the fact that our future might be fewer surveys, or at least less emphasis on surveys as the centerpiece of research. It's harder for me to believe that the empirical part of what we do, the ability to evaluate evidence in a systematic way, and the need for disciplined thought about a client's problem will go the way of the dodo. We probably need more craftsmen, not fewer. But the craft surely is changing.

Finally, the real issue?

This month's Research Business Report has two very interesting and related pieces. In one the editor takes the ARF ORQC to task for their somewhat languid reporting on the results of their Foundations of Quality initiative as well as the usefulness of their findings, or at least those findings that have been released to date. (In case you've not been paying attention, the ARF has spent a ton of money doing basic research on research aimed at improving our understanding of some of the long-talked about quality concerns with online panels.) At the risk of oversimplifying, I would characterize the ARF results to date as reassurance that the principal panel quality issues the industry has been most worried about over the last few years—fraudulent respondents, multipanel membership, people taking the same survey more than once, high levels of satisficing, etc.—are real but controllable, either by the panel supplier or the researcher.

Unfortunately, in the process of answering what we thought were the big questions they have raised an even bigger one. Here is one bit of the ARF data that has made it into the public domain. When comparing the incidence of smoking in survey results from 17 panels to benchmark data using traditional probability-based methods they found that (a) the three probability-based methods produced similar results while (b) the variation across panels varied by as much as 14 points. This should not surprise us. Panels are all very different in how they recruit their members and the way in which they manage those members over time. We can control the demographics, at least at the sampling stage, but the mix of attitudes and behaviors of panelists will vary substantially from panel to panel based on a wide range of things (e.g., where they were sourced, the appeal that was used, the incentive scheme, the survey topics they offer, etc.) that we still don't understand very well. In the end, the results we get for any given study are highly dependent (and mostly unpredictable) on the panel we use. This is not good news.

Which brings us to the second piece, the RBR lead story. For about the last 18 months Steve Gittleman has been doing something he calls, "The Grand Mean Project." In a truly prodigious effort, Steve has amassed survey data covering something like 90 panels worldwide and has been using those data to study variability among panels. He argues that with the move away from probability-based methods and the consistency that comes with a rigorously applied scientific approach we have lost our anchor to reality and with it any hope of using panels to get at "true values" for survey variables. Rather, the best we can hope for is that the same survey administered to the same panel multiple times will produce consistent results. Administer that survey to another panel, and you likely will get different results. Steve believes that he has developed a method that measures panel consistency over time and that you can use that method to identify the most consistent panels. In essence he is making an argument others have made to the effect that bias is not necessarily a bad thing as long as we know it's there, it's consistent over time, and we have other sources of information on the same topic that can help us correct it.

A little over ten years ago, Gordon Black, founder of Harris Interactive, asked "whether findings from huge samples of Internet respondents, coupled with sophisticated weighting processes, are as accurate as anything done on the telephone or door-to-door." More and more it seems like the answer is no.

More thumbs down on eye candy in surveys

The current issue of Inside Research has a short note on some work done at DMS looking at respondent reactions to rich text and graphical interfaces in Web surveys. It's hard to go to a MR conference these days without seeing someone present a passionate argument for the use of Flash or various kinds of interactive gadgets as a way to increase respondent engagement and therefore improve data quality. Readers of this blog know I am skeptical. Nonetheless, I tracked down the presentation from Chuck Miller at DMS.

It turns out to be an interesting piece of research. (There is a video of Chuck describing his research here.) He took sample from a variety of sample sources and allocated it out to three treatments: (1) forced into a plain text survey; (2) forced into a rich text survey; and (3) allowed to choose either plain text or forced text. Quotas were used to force the demographic distributions. The key findings:

• Three out of four respondents in the choice treatment chose plain text.
• Choice seemed to be driven by lifestyle (e.g., lots of time online, electronic gadget geek, etc.) rather than demographics. Rs choosing rich text also reported taking lots of surveys.
• Rs in the rich text group gave higher ratings of the survey experience than those in the plain text group, but mid-survey terminations in the forced rich text group were higher than the forced plain text group.
• For the most part, there were no differences in response data itself except for the slider bar questions where Rs using the slider bars reported significantly more time watching TV, listening to radio, and reading magazines than those using standard radio buttons.
• There were no significant differences across a range of data quality indicators except for straightlining which was higher in the plain text treatment. However, I have always felt that the standard radio button grid lends itself to straightling while many of the rich media answering devices do not, but that doesn't mean the latter is yielding more thoughtful answers.
• Overall, the best approach seems to be to let people choose their interface, but the practicality of that with today's software tools is questionable.

I am reminded of research, now several years old, that looked at slider bars and sorting gadgets back when we first figured out how to incorporate them into surveys. That research mostly reported that these devices caused us to lose more respondents (either because of technical issues, confusion, or not liking the device), generally took longer to answer, and produced the same results as conventional answering devices.

This whole debate, if there is one, is symptomatic of a basic problem in the evolution of online research. Someone gets an idea, sometimes builds a business around it, and advocates it loud and far as the next big thing with no real data or facts to substantiate the claims. Because it's cool or fast or less expensive we buy into it, only to eventually find an emperor with no clothes.

We really need to do better.