The Survey Geek

Let's hear it for conventional wisdom!

The post begins, "Contrary to conventional wisdom, U.S. consumers age 65 and older are most likely to shop online (77%). Consumers ages 45-64 and ages 30-44 follow at 71% with those ages 18-29 at 70%, a new report finds."  So 77%of persons aged 65 or older shop on line?  That's especially astounding because by most estimates only about 45% of persons 65 and older are online.  And presuming this study was done online (the post doesn't offer that detail) we need to also accept the reasoning that seniors on somebody's online panel are just like all online seniors in their use of the Internet.

Someone (The Cable and Telecommunications Association for Marketing) is actually selling this stuff!  One only hopes that nobody is buying it.

Have we reached a turning point in online research?

 

I am starting to believe that 2009 may be a watershed year in the evolution of online research. At least three important developments stand out.

First, after several years of widespread concern about panel data quality the industry appears to be arriving at something approaching a consensus about what constitutes a "good" panel, the steps that a researcher should take when choosing a panel provider, and the best methods for post survey cleaning to produce a useful dataset. The themes one sees unfolding in the ARF's Quality Enhancement Process are similar to those in other industry initiatives such as the ESOMAR 26 Questions and RFL's Platform for Data Quality Progress. Now we need to execute.

Second, there is the gradual recognition among the more thoughtful people in our industry that the academics were right all along in maintaining that while there are many challenges to conducting true probability sampling, when done right it yields more accurate estimates than the non-probability methods used by all but a few panels. If you've not been following the public debate on this you can pick it up here. No one is suggesting that we should stop using panels, only that we make wiser choices about when to use them, make more restrained claims about their representativeness, and improve our ability to interpret their results.

Third, the research industry is finally coming to realize the preeminent role of questionnaire design in research both as a guarantor of survey data quality and a means of preserving the public's willingness to do surveys. We will continue to have arguments about the best ways to present survey questions, but we know with certainty now that shorter and more engaging surveys are essential to the continued survival of online survey research. Once again, the key will be execution.

Taken as a whole these three developments can at last migrate online out of its "wild west" period and into something resembling a mature research method. Challenges remain and there are many who have yet to catch the wave, but this is most progress I think we've seen over the roughly 15 years since online first emerged. It's been a long time coming and a welcome sight.

Sometimes it’s the little things

One of the things that has always irked be about online is the poor graphical design of many of the surveys that find their way to my inbox. At one point we established a folder on our corporate network where people could dump screenshots of the especially bizarre. We called the folder, "Hall of Shame." The web offers us a vast palette of color and interactivity with which to design surveys, but as often as not it seems that people just make a mess of it.

Over the years we have done experiments with some aspects of design including background color, screen size, progress indicators, and location of the navigation buttons. Some of this research is reported in Mick Couper's excellent book, Designing Effective Web Surveys. We have used the results of those experiments along with what we can find in the web usability literature to create the corporate standards that we apply to every survey we do. Oddly enough, no element of those standards seems to be getting more questions from clients these days than placement of the Next and Back or Previous buttons.

The natural positioning in many people's minds is to mimic the Back and Forward buttons on the browser: Back on the left and Next on the right. But our research shows that it works best for respondents if we do the reverse: Next on the left and Back on the right. We've run two experiments and the results have shown:

• Surveys can take longer to complete with the Next button on the right and the Back button on the left.
• Respondents terminate at a slightly higher rate with the Next button on the right and Back button on the left.
• Respondents terminate at an even greater rate when we don't give them a Back button at all.
• If you put the Back button on the left respondents will use it more, probably by mistake.

Now I've also just stumbled across some research in the form design literature by web form design expert Luke Wroblewski that confirms placement and adds a new dimension I'd not thought of: color. He is trying to understand the best way to present what he calls "primary action" and "secondary action" buttons. In web form design this typically means Submit and Cancel. The results seem to be to be equally applicable to Next and Back. His findings:

• People seem to work with forms most efficiently and with the fewest errors when the Submit button is on the left and the Cancel button on the right.
• The worst design—the one that slows people down and causes the most mistakes—has the Submit button on the far right and the Cancel button on the left.
• Using color to distinguish the buttons (green for Submit and red for Cancel) seems to work very well. Better still, make Cancel a link rather than a button.

Now this may seem like a silly little issue and not worth getting all worked up about, but as we finally begin to recognize the importance of creating an hospitable and easy experience for survey respondents these little things do matter.

Might Web researchers be developing a conscience?

Amongst all the talk in Montreux about corporate social responsibility was some interesting discussion about how Web survey researchers also might develop a little social responsibility of their own.

First, there is Research-Voice.com. This is the brainchild of Kees de Jong (now CEO at SSI) who for the last couple of years has been championing what he calls a "respondent preservation initiative." The concept is simple. Without respondents there can be no surveys and so it is in the best interests of the industry to recognize that they are not a commodity to be used, abused, and cast off when we are done with them. We need to put respondents front and center and treat them with respect in all of our interactions with them. I certainly have spent some time in this space ranting about poor survey design and its impact on both respondents and response quality. It's a serious problem and Research-Voice tries to draw attention to it as well as encourage research-on-research aimed at improving the respondent experience. Check it out.

Then there is the "Pause for a Cause" initiative launched by the CMO Council. The idea here is to create a panel in which incentives go not into the respondents' pockets but to charities of their choosing. The goal is to redirect at least some of the money spent on online research into charitable donations. Needless to say, this idea has received a less than warm welcome by at least some companies in the panel business. Researchers also are expressing some concern that the motivations to join this panel (you can check it out here) might result in a whole new set of biases in its samples. A colleague has suggest that the better approach might be to offer the infrastructure to channel incentives to charities to all panel companies rather than launching a whole new panel composed of people with a sense of social responsibility.

I am rooting for Research-Voice to succeed but share the worries about bias in Surveyforgood. I like my colleague's suggestion about repositioning it as a service rather than a separate panel.

Some marching orders from Neils

Regular readers of this blog (if there are any) know that I have some genuine enthusiasm for Research 2.0 or Connected Research or whatever we call MR in the age of Web 2.0 and social media.  Yet I still find it largely undefined, mostly a collection of buzzwords still awaiting a real definition as a research method.  Niels Schillewaert at Insites Consulting in Belgium is doing some of the best thinking of anyone about what this future looks like.    I've recommended his stuff before and I follow that up here with the suggestion that you have a look at this video of one of his recent presentations.  I could nitpick with a couple of things he says, but not with the central thrust and certainly not with what seem to be his two central points: (1) we need to think long and hard about what all of this means and (2) begin defining the the skillsets and the tools we need to do research in a fundamentally-changed world.  A tall order for someone who has been doing what I do as long as I've been doing it.