We are crafting a response to an RFP that contemplates transitioning a mail study to the Web. I've been asked the question, "What kinds or problems might we encounter?"
Tough question. I am hard pressed to think of studies I've seen that take a good, systematic view of the issue. So mostly, I am guessing. For what it's worth:
- Expect the response rate to drop. At least 30 percent of US households do not have Internet access but I expect all of them have access to a pen or a pencil. Since this is customer sat work for an insurance company the proportion without Internet access is probably smaller than 30 percent, given the SES and age bias associated with Internet access. More serious is the natural affinity that people seem to have for paper and pencil as opposed to Web. I have seen a couple of studies which let the respondent choose the mode and mostly they seem to choose paper. Web is just too much trouble for a lot of people. For this reason it may make sense to provide a mail questionnaire as an option at the time of contact.
- Expect a different demographic bias. I would expect there to be significant demographic differences between those who have previously responded by mail and those who will respond by Web. In a general population study this might not be a major problem because one could correct with demographic weighting. In this case, however, the demographics of the sample frame (the client's list of customers) may not be known and so there may be no clear target to which the survey results can be weighted.
- Don't expect mode effects of the magnitude we sometimes see in phone to Web conversions. Since both are self-administered in a visual mode the normal sources of mode effects are not at issue.
- Expect to redesign the questionnaire. There is a tendency to make the Web questionnaire an online version of the paper document, all the way to look and feel. The rationale is data comparability across the two modes. But a redesigned Web questionnaire can make dramatic improvements in data quality by including edits, validations, and automated skips that virtually eliminate the errors made by respondents with paper questionnaires.
Of course, your results may vary so parallel testing is essential. If you are lucky, the differences may be insignificant and can be ignored. More likely you will see some of what I have described above but you also may be able to correct for them over a couple of rounds of experimentation.