Previous month:
January 2009
Next month:
March 2009

Posts from February 2009

A sign of the apocalypse?

I am running for Chair of AAPOR's Standards Committee.  I'm doing this because I view AAPOR as the premier organization of the survey research profession, but they have stayed on the sidelines so far in what I view as the biggest challenge to the profession since the 1936 presidential election.  To appeal to a much overused analogy, the decline in respondent cooperation, the rapid growth of wireless only, and the dramatic emergence of online research have created a perfect storm.   We need the smart minds and the scientific orientation of AAPOR to help us work through it. 

Of course, there are those who no doubt believe that electing me to chair Standards would be to ask the fox to guard the chicken coop.  These folks can for now take solace in the fact that I have run for AAPOR posts before and I have proved to be, in the words of a friend, "the Harold Stassen of AAPOR."  It takes either a real optimist or a complete fool to believe that somehow this time will be different.

For the record

I had an email yesterday from a friend and colleague telling me that MarketTools was sending an email around intimating that I somehow had blessed their True Sample product. The claim is a bit of a stretch but I'll save that for another day. More importantly, the friend went on to say that she had seen Andy Morrison's interview in Research which was a little tough on online panels. Given that she and I have worked together on a number of panel quality initiatives she asked me if I would update her on my latest thinking about those and other similar initiatives. Hence this post.

Andy's interview was all about the scientific perspective which says that there is simply no way using the standard statistical tools we all know and love to accurately estimate population parameters from a sample drawn from an online panel. The sample frame—the panel—is inherently biased. To begin with, not everyone is on the Internet and then not everyone on the Internet chooses to join a panel or, for that matter, the specific panel we might use to do a specific bit of research. The differences between people on the panel and the population at large go well beyond the demographics we typically use to describe "representativeness" and include a whole range of complex attitudinal and behavioral characteristics that thus far have escaped measurement. Probability sampling stipulates that the sample frame be a full list of everyone in the population of interest or if a subset (as in a multi-stage designs) systematically arrived at. Opt-in panels fail this test. There simply is no legitimate way using standard statistical methods to project estimates produced by a sample drawn from a panel back to the population as a whole, unless, of course, the panel is recruited using probability methods and includes the offline population.

All of which is not to say that we can't generate great insights and do good research with opt-in panels. The challenge is to understand the bias involved and what impact if any it may have on the business problem we are trying to solve. In other words, we need to be sure that panel research is the best method for the problem at hand given the client's problem, the target respondent, and the time and money available.

The panel quality initiatives now underway are something all together different. I see them as coming out of the TQM tradition rather than a scientific or statistical tradition. Ever since Kim Dedeker uttered her now famous "Two surveys a week apart by the same online supplier yielded different recommendations … I never thought I was trading data quality for cost savings" the industry has been focused on figuring out what went wrong and how to fix it. Viewed in TQM terms, we have too much variation in output (the survey results) and the way to fix that is to reduce the amount of variation in the inputs (everything that happens from panelist recruitment through to completion of a survey).

There is now a lot of research on research aimed at understanding best practices in recruitment, validation, panel management, sampling, survey design and so forth. Even as we learn more not everyone will agree on the best way to do these things, but once an individual panel company chooses its approach, standardizes on it, and operates consistently within those standard approaches survey results from that panel will be more predictable, replicable, and, in the eyes of clients, more valid. Initiatives like ISO 26362 and the ESOMAR 26 Questions take the next step by creating transparency so that an educated consumer can make informed choices about which panels to choose for a specific study.

I think all of this is absolutely terrific and the industry deserves enormous credit for how it has responded to what is admittedly a self-inflicted wound. I think research using panels will become much more reliable and predictable as a result of this industry-wide initiative. But it will not mean that all panels are the same or that the same study executed with two different well-managed panels will yield the same results. Nor will it mean that we can make stronger claims of representativeness or describe our results in terms of "true values" in the statistical sense.

Online has given us both a new set of research methods and an enlarged responsibility for more nuanced interpretation. It is more important than ever that we interpret results in the broader context of other available information on the same and similar topics. Fortunately, we live in an age rich with detailed and easily accessed information and a broader set of research techniques than ever before. Our challenge is to use them wisely and in ways that help our clients to understand their businesses better.

Mobile Research Conference 09

Reg has kindly allowed me to act as a guest blogger. I'm in London at the (first?) Mobile Research Conference, put together by Global Park. About 75 people in attendance, mostly suppliers and academics. Today was the first day and we had 6 sessions ranging from broad keynotes to case studies of specific research projects conducted on mobile platforms. My broad observations at this point are more about what isn't being said than what is:

  1. A lot of emphasis is placed on the advantages of mobile platforms as "personal" in a way that web surveys on computers are not. Not honestly sure I understand this. I understand that people have a different relationship with their phones than with their PCs, and that PCs are often shared, but that doesn't mean computers aren't personal, and in any case I'm not sure what practical advantage a "personal" device has for research. I certainly see some of the practical disadvantages – principally that if a research invitation is greeted as spam, it's likely to raise more ire on a phone than in email on a computer.
  2. Research using SMS and web surveys on a phone browser are the first modes I'm aware of that have cost implications to participants (other than the value of their time). Six questions posed by SMS can cost the participant upwards of $2. In countries with high rates of pay-as-you-go mobile plans, the cost can be quite high. This has clear implications for how we provide incentives, but I suspect it has other implications as well, which haven't been thought through well.
  3. There is a huge need for sample sources. There is no frame of mobile phone numbers and no real equivalent of internet access panels as a driver of mobile phone research growth yet. The panels that exist are subset from existing internet access panels so its coverage error compounded with coverage error. The panel providers who spoke typically saw 30% uptake among existing panels to participate in a mobile panel, but surprisingly the common remark is that their mobile panels are underused.
  4. Questionnaire length limitations are and probably always will be a feature or limitation of this mode. We need to seek out more creative ways to use the mode beyond just asking questions. Sweetening responses with other forms of data (location services, photos) helps but it also turns the mode into a vehicle for discussions that are semi-qualitative in nature – very cool but not fitting very well into the model for efficient research that the industry relies on now.
  5. Every presentation focused inordinately on how quickly people respond. It reminds me of the early panel days when speed was touted as an end rather than a means to an end. I am interested in compressing time to results, but with the exception of certain polling and media research context, I don't think that getting responses in 2 hours rather than 2 days really addresses a client need.

Apologies for any typos. I'm trying to embrace the casual nature of the medium. Those interested in going a step further can consult the conference's tweets at \mrc09.

"The world is not so flat after all."

I saw this quote a while back and being a Tom Friedman fan it stuck with me.  It was attributed to Pankaj Ghemawat, but after reading his book (Redefining Global Strategy) I'm not sure that he ever actually said it precisely that way.  The central premise of his book is that the world is only "semiglobalized" and will remain so for some time to come.  He sees that perspective as an important way to help companies "resist a variety of delusions derived from visions of the globalization apocalypse: growth fever, the norm of enormity, statelessness, ubiquity, and one-size-fits-all."

At the center of the book is a framework Ghemawat calls CAGE, and acronym built up from the the four kinds of differences he believes companies must consider in their global strategy: Culture, Administrative/Government, Geography, and Economy.  These differences mean that global ventures pretty much mandate a country-by-country strategy both for selecting countries for expansion and configuring operations within those countries.

Doing research globally is no different.  There is a strong tendency to use the same research methods elsewhere in the world that we use here in the US.  Online is a clear example.  Despite low Internet penetration, well-known biases in Internet use, and uncertain impacts of culture on survey response there are online panels growing up in countries like China and Russia with no shortage of research companies using them. We tend to ask the same questions, use the same scales, and present the same kinds of exercises that have become commonplace in the US and assume they will work the same in other countries. I'm not sure that anyone knows how well we as an industry are doing or how useful the insights are that we are delivering.  Our own experience tells us that it's tough work and for all of the reasons Ghemawat suggests.

Clients have enormous interest in emerging markets, and that interest is driving a dramatic increase is research in Asia and in Latin America, the two fastest growing regions for MR.  Expect this trend to not just continue, but intensify.  We had better get our act together.

Sweating the small stuff

Andrea Widener asked me a question today about our use of underscore for selective emphasis in Web surveys. We used to use blue. Andrea's query made we wonder what the current thinking is on the topic. For this I turned to Mick Couper's Designing Effective Web Surveys. Now I should divulge at the outset that (1) Mick is an old and good friend, (2) he mentions me in the acknowledgements, and (3) I praise the book on the back cover. Thing is, I meant what I said.

At any rate, I was curious about what Mick had to say on the topic. He runs through all of the options. He warns that UPPER CASE is usually a sort of visual search tool in documents to help you find things and generally does not encourage reading all the text. Italics are hard to read, especially if there is a lot of it on the relatively low resolution of a monitor. As for our old standard blue, he quotes usability guru Jakob Nielsen, "On the Web blue text equals clickable text, so never make text blue if it is not clickable." And then Nielsen goes on to warn about using underline for anything that is not a link.

As for color, well he goes after that, too. He points out that many colors—like blue, green, and yellow—are inherently weak compared to black, and so I suppose they might de-emphasize rather than emphasize. He also warns that color has meaning and therefore might influence how someone interprets a question. And, of course, some people are color blind.

In the end he recommends "bold, italics, or UPPERCASE, in that order. I guess we have some rethinking to do.

He also goes on to warn against a common problem: overuse. Remember, it only stands out if it is really different from everything else on the page. After all, it is supposed to be selective.