The Survey Geek

A Caution About Video

This morning's email had a link to an interesting study by one of the foremost experts in Web usability, Jakob Nielsen.  The headline: "Talking-Head Video is Boring Online."  You can read it yourself here:  http://www.useit.com/alertbox/video.html .

The bottom line: his eye tracking study showed that even in the short video Nielsen tested (24 seconds) the test subject's eyes often wandered the page looking at and reading other stuff as the talking-head droned on, suggesting that she was quickly bored with the video. To extrapolate to my own reactions to video on the Web: the Internet is about lots of content speedily delivered and video is inherently slow and ponderous. 

But what are the potential implications for surveys?  Two come to mind.

First, there would seem to be a caution here for those who dream that some day Web surveys will be done with "virtual interviewers," that is, video of a human interviewer asking the questions.  This certainly has been talked about with the added argument that you can control potential interviewer effects by matching interviewer characteristics (like gender, age, and race) to the respondent.  None of this has been extensively tested and it surely needs to be before it's put to use, but this study suggests that this may be one more thing people do in Web surveys simply because they can, not necessarily because it adds anything to the respondent experience or improves data quality.

Second, there may be a more general caution about using video of any sort in Web surveys.  Do respondents watch it?  For how long?  How closely?  What kind of multi-processing might they be doing?  If nothing else, this study would seem to suggest that keeping it short is a good idea.

This is a topic on which I expect we will hear much more over the next few years.

What were they thinking?

I recently received a postcard from IPSOS to go online and complete a survey for BP.  The instructions were on the back of the postcard.  Click here to see it.

I could nitpick on this and point out that unless you know the BP logo you might not understand that this is a survey about where you buy your gas, but we'll let that go.  The real problem is getting to the survey.  First, they give you a tough URL to type into your browser.  Most people are familiar with URLs that start with "www" rather than the more mysterious "http://" so that is tough enough.  Second, they' give you a URL with a somewhat unconventional domain name with the "-" character in it.  So just typing the URL correctly took me a couple of tries.

But the real puzzle unfolds when you get to the site and have to enter your ID and Password.  The postcard says that this information is included on the front.  You can see the front here. Somewhere on there is an ID and a Password, but it sure was not obvious to me where.  After several tries of different combinations I discovered that my ID is 91646492 and my Password is SURVEY.   

You have to wonder about the cumulative effect of all of this on survey participation.  I stuck with it because I'm a survey geek.  But who else will?

How would you rate your health right now?

This is a question we are fond of asking, and it has emerged as a major focus as we look to migrate healthcare-related telephone studies to the Web. The full question looks like this:

How would you rate your health right now?

1. Excellent
2. Very good
3. Good
4. Fair
5. Poor

When we did you full-fledged test of RDD sample vs. an SSI sample back in 2004 here is what we got even after weighting to account for demographic differences:

These data suggest that the telephone respondents may be healthier than Web respondents, or at least they say they are.  And while it sort of evens out if you combine "Excellent" and "Very Good" the 12 point gap on "Excellent" is troubling.

These data also bear a striking resemblance to data from a similar experiment conducted at RAND and reported in Schonlau et al., "A comparison Between a Propensity Weighted Web Survey and an Identical RDD Survey," Social Science Computer Review, 2004, 22, 128-138.

Here are their data and our data combined with the phone results in blue and the Web results in yellow and orange.  Pretty interesting.

Moving forward to 2005 we are looking at data from a Web experiment we are doing on a study of health plan choices by employees of a major manufacturing company.  Here are those data:

The distributions are somewhat different but the general finding of poorer health reports by Web respondents holds up.

So what's up?  Well, when I presented the PULSE results at a conference in 2004 no less a luminary that Don Dillman said, "Reg, you have a mode effect.  People shy away from choosing extreme values in the visual mode."  Could be.  There is some evidence of that in other research.  But in the 2005 health plan study we also asked about skill using the Internet with the exact same scale as our health question and displayed in identical fashion.  Here are those results:

Web respondents here seem to have had little difficulty choosing the extreme response, although the percent choose "Fair" is interesting.

I don't believe this is about "extreme responses" at all.  I think it's social desirability.  People want to present themselves in a positive light to other people (such as telephone interviewers) but that goes away in the context of the relative anonymity of a Web survey.  There are lots and lots of studies that have shown people will under report really sensitive behaviors like abortions and drug use in interviewer administered surveys.  For these kinds of issue self-administration is the way to go.  I'm surprised to find that respondents apparently view their health status as sensitive, but the data are pretty persuasive if not compelling.