The Survey Geek

Online MR Growth Continues

Interesting reading in the January issues of still another trade rag.  This time it's Inside Research and the topic is the rapid growth of online research over the last 10 years, from just $4 million in 1996 to a reported $1.4 billion in 2006.  The projections for 2007 are for 15 percent growth to $1.6 billion.  For those of you who don't follow these things closely, that's 38.5 percent of total MR survey revenues expected in 2007.  Impressive for sure, but not exactly fitting the recent RBR headline, "Online leaving CATI in the Dust." 

There are two other interesting factoids in this piece.  The first is that this growth has been driven almost completely by panels with an estimated 87 percent of online research using private or public panels as their sample source.  Offline recruiting--a.k.a phone or mail to Web--comes in at an insignificant 1 percent.

The second is the type of research being done.  Concept and product testing accounts for about a third, sales tracking and U&A together constitute another third, followed by smaller amounts of advertising and brand tracking, customer sat, copy testing, and so on.

The interesting question posed by IR at the conclusion of the piece is the likely impact of emerging client concerns about panel response rates and the validity on online survey results.   IR is not sure and neither am I.

Predictions for 2007

Not mine, but those of about 50 "esteemed industry players" reported in Research Business Report, a trade rag that follows the MR industry.  Once I was able to put aside my disappointment from not being an "esteemed industry player" I found the reading interesting. As you might expect, the predictions are all over the board but there are some themes:

1. Several people see text mining of online content (e.g., blogs and bulletin boards) as major new avenues for understanding the mind of the consumer, especially for groups that spend lots of time online (like teens).
2. The notion of "a flight to quality" is raised by several people, most notably in the context of expected client concerns about respondent cooperation and panel quality.
3. As in years past, there is discussion of "the Wal-Martization of MR."  Some see it getting better; others see it getting worse.
4. Lots of people see more offshoring.
5. And lots of people see ethnography as increasingly important, especially with the application of new technologies--video, webcams, etc.
6. There are a fair share of mentions about Web 2.0 but it's not clear that anyone has a fix on how it will impact our business beyond extending the online communities that some people see as the logical successors to panels.

But my favorites are always the people who go out on limbs to predict the emergence of new, almost magical technologies.  A few favorites from this year:

• "In less than seven years, a data collection medium that has not yet been widely adopted or perhaps even tried, will surpass phone and face-to-face methodologies."  (Dean Wiltse, Perseus/WebSurveyor)
• "By the end of March 2007, one company will have at least 200,000 homes sending second-by-second individual commercial engagement data directly to advertisers . . ." Lee Weinblatt, The Pretesting Company)
• "In the next decade, a never-before-seen MR phenomenon will serve as a historic marker for this industry: an online survey so stunning in its power to engage (that it) will be shared by millions of surfers the same way powerful ads are shared today. (Angus Reid, Angus Reid Strategies)

I can only assume that these guys have something specific in mind and it probably is under development in their respective companies as I type.  But before you invest, perhaps see my earlier Jan 2 post on Innovation: Hits and Misses.

The Professional Respondent Problem

One of the best papers at the ESOMAR Barcelona conference was presented by Brian Fine and his colleagues from AMR Interactive (an Australian research company) titled, "Attitudinal Differences: Comparing People Who Belong to Multiple Versus Single Panels. " I use the word "best" because it was one of two papers so designated by the Programme Committee. And I use the term "professional respondents" because increasingly that's the term being used to describe people who belong to multiple panels and who do lots of surveys. 

The paper looks at demographic, behavioral and attitudinal differences among respondents across a continuum that starts with CATI then goes from single online panel membership to several categories of multiple panel membership.

One unsurprising finding is that online often produces different results than CATI.  The paper argues that propensity weighting may be able to solve this, although I should note that the AMR Web site references a partnership of some sort with Harris Interactive, center of propensity weighting.  That said, CATI/Online differences are not really the theme of the paper. 

The central theme is the effect of single vs. multiple panel membership and the findings are striking.  For example:

• Single panel members are more likely to be older, male, educated, working full time and owning a home.
• Single panel members gave higher ratings on bank reputation and lower ratings on airlines.
• Multiple panel members drink less wine and invest less, but are more likely to smoke and own pets.  They also are more likely to read magazines and own high-end technology products.

At the first panels conference in Budapest a paper from Survey Sampling, Inc. reported that attitudes on critical questions like propensity to buy a product varied by survey taking experience.  The Fine study provides another perspective that would seem to support the SSI finding: people who belong to multiple panels and do lots of surveys are different attitudinally and behaviorally from those who belong to just one panel and do fewer surveys.

Unfortunately, Fine and his colleagues don't take us to the next step and help us to understand how we can protect or correct.  They point out, for example, that even setting limits on survey participation with your panel provider doesn't help because your panel provider does not know how many surveys the member may have done with other panel companies.  And, of course, there is no obvious weighting magic that evens all of this out.  The paper ends with a noble but nonetheless frustrating call for more research.

So what's a researcher to do?  Well, at a minimum we should be routinely asking online respondents about the number of online surveys they are doing.  At least we can start to measure the problem in our own work.  This at least will allow us to understand a little more about our online samples and perhaps help us to understand any anomalies we see in our data.  And we should be pressuring our panel suppliers to do more to identify multi-panel members and at least flag them if not eliminate them from their panels.  But it seems that a clear, effective solution is still a ways off. 

We live in interesting times.  Perhaps, too interesting.

The Future of the Survey Research Industry

This is the title of a recently-released report based on a survey of CASRO members conducted in August and September of 2006.  (The full report is here:  http://www.consultcambiar.com/news/The%20Future%20of%20The%20Survey%20Research%20Industry.pdf .) As its title suggests, the report is focused on how CASRO members see the future of our industry, both in terms of the key challenges we face and the ways in which client needs and expectations may change.

The challenges seem to fall in three broad areas: respondent cooperation, technology, and staffing.  The cooperation issue tops the list with 72 percent of respondents identifying it as a major issue for the next five years.  It is hard to think of anything more pressing than figuring out how best to convince people to tell us what our clients need to know to make informed decisions.  The principal concern in technology revolves around the ability to make the needed investments to stay competitive, both in terms of collecting and processing survey data as well as delivering it to clients.  The third issue of staffing breaks down into several different areas with over half of respondents concerned about some combination of research skills, ability to sell, and account management. 

A major issue driving the concern about staffing is the expectation that clients in the future are going to want a lot more from their MR partners in terms of business knowledge, consulting, and even strategic planning than they have in the past.  The skills that CASRO members believe will be important going forward reflect that expectation.  They include having knowledge of and understanding client business issues, the ability to consult with senior management, and the ability to tell a story.  But topping the list is the ability to integrate data from different sources, something that will be increasingly important as we try to deliver more holistic solutions to clients than we have tended to deliver in the past.  For a variety of reasons, the longstanding tradition of designing and conducting a primary research project and then reporting the results may well give way to reports that rely on a variety of primary and secondary data sources.  The ability to absorb, analyze,and synthesize all of this into a compelling story for the client will be a key skill.

All that said, it is interesting to look at the specific roles that respondents say will comprise a successful research company in 2016. They are project managers (80 percent), analysts (78 percent), account managers (77 percent), research managers (74 percent), statisticians (70 percent), IT specialists (64 percent), and consultants (64 percent).

There are also a few interesting disagreements:

• Fifty-seven percent disagree that in order to be competitive they will need to offshore some of their processes.
• Fifty-nine percent also disagree that they need to have all of their functions in-house to ensure data quality and integrity.
• The report says:  ". . . CASRO members emphatically do not believe that all research will be panel-based in ten years!"  The precise distribution on this item is not reported.

And finally, just 45 percent of respondents say that they would recommend survey research to their children as a career.  Do what you will with that one.

Innovation: Hits and Misses

The research industry has seen more than its share of innovation over the last five years with much of it related to the Internet.  The December issue of Research World includes an interesting little piece by  Ray Poynter of Virtual Surveys in the UK in which he sizees up eight of these innovations to identify the winners and losers.  His winners:

• Online access panels.  Now there is a no brainer!
• MaxDiff scaling.  Shorthand for Maximum Difference scaling.  It's a technique that tries to solve two thorny  research problems: cross-cultural differences in use of scales and highly skewed scales like those in customer sat studies.  You can find a better explanation here:  http://www.sdr-consulting.com/article19.html.
• Video-based ethnography.  This is more about packaging and reporting than it is the actual research.
• WOM research.  WOM being short for word-of-mouth.  Think of all of those new companies with the word "Buzz" in their name.

And now for the losers:

• SMS or text-based research.  Is anyone surprised that not a whole lot of people like doing surveys via text messaging on their cell phones?
• Online focus groups.  The advantage--lower costs and less travel--could not overcome the key disadvantage of reduced interaction.
• Virtual shopping.  Looks cool but real insights and advantages over traditional surveys have been hard to come by.
• Neuromarketing.  We are talking brain scanning here.  Expensive and slow.

Ray classified innovations by asking four critical questions:

1. Is the solution a significant improvement over current practice?
2. Does it meet an unmet need?
3. Do buyers perceive additional value?
4. Is the service available form a range of providers?