The Survey Geek

ESOMAR 3D

This is an unabashed plug for an upcoming conference: ESOMAR's 3D Digital Dimensions 2011 to be held October 26-28 in Miami. (Disclosure: I am chair of the Programme Committee.) This will be the latest in a series of thematic ESOMAR conferences that more than any other has charted the evolution of online research from panels to social media to mobile.   It is one of the few conferences that requires a written paper which I have always found is a sure fire way to elevate the quality of the research that's presented.  It is the seventh in the series that began in 2005 in Budapest and the third to be held in North America. Like its predecessors it will be truly international with presenters from 11 different counties and features keynoters Phillip Sheldrake and Nathan Eagle, two speakers with especially interesting perspectives to share. The program deals with the three main streams of online research: panel, social media, and mobile. There also will be preconference workshops on social media, online ethnography, MROCs, co-creation and mobile.

You can check out the full program here. Hope to see you in Miami.

Two cheers for mobile: MRMW 2011

I spent two days this week in suburban Atlanta at Market Research in the Mobile World 2011 (MRMW 2011), a conference as its name suggests (almost) totally focused on mobile research. It had an imposing agenda with a whopping 29 presentations all in the same room over two days. The presentations were grouped into four "modules" and at the end of each one we had an "activation" session (get it? mobile? activation?) to develop actionable recommendations for the industry

The mobile hype was thicker than the Georgia summer air outside with lots of dazzling (although sometimes questionable) numbers on mobile growth and skyrocketing smartphone penetration along with assurances about people's willingness, nay eagerness, to serve up the intimate details of their personal lives in unprecedented ways. (In one presentation we were shown a smartphone self-portrait of a smiling couple in bed as a part of an in-the-moment qual study about how they spent the morning when hubby stayed home from work and baby slept in.)  There was an evangelical revival meeting flavor to the affair with an added element of surrealism given that we were talking about using mobile to probe deeper into people's lives at the same time Rupert Murdoch was being grilled in London by the Select Committee. When one doubter worried aloud if we were moving too far too fast and needed to do more validation studies he was quickly denounced, but then just as quickly reassured the mob that he "lives and breathes this stuff." Me, I kept my head down.

But there were moments of genuine enlightenment as well, at least for a mobile ingénue like me. There was a nice presentation on the promise and the technical challenges of geolocation research; an entertaining and very informative presentation on HTML 5; some good examples of effective uses of SMS, especially in developing countries; a debate of apps versus browser-based approaches; some intriguing qual methods involving mobile and a nice sampling of the different kinds of mobile implementations both currently being used and under development. There also were a couple of presentations about social media that in this setting seemed almost retro. The envelope was being pushed and pushed hard.

A few observations:

• Whether it's ready for prime time or not mobile research is not just here to stay, it's poised to grow very quickly. It will be messy and there will be lots of problems. The key question: when will it start to command a significant part of client research budgets and move from the boutiques and the startups to the mainstream agencies? Those agencies, the software developers and the panel companies are all hedging their bets so that when/if it does take off in a major way they are ready.
• Clients are going to have to think about research differently if they want to deploy mobile on a broad scale. One of the major challenges with online has been questionnaire length, much of it due to the need to drive the advanced analytics that have been one of the major stories in MR over the last decade. So, for example, is customer sat going to devolve to all NPS all the time? One of the counter arguments here is that the richness of detail that clients are accustomed to getting from surveys will now come from behavioral data and mobile surveys may be used more to fill in the gaps. But that's a big shift and a hard one that will require clients to fundamentally rethink their research processes.
• Mobile inherits all of the problems of online panels—coverage error, sample bias, professional respondents, and a shortage of willing respondents. Novelty may bring some new people into the mix but as the use of mobile grows and becomes routine there is no reason to expect that we won't be stuck in the same place we are today with classic online.
• Privacy is a much bigger problem than the mobile evangelists want to admit. It's not just the concern of a few misguided politicians. We already are seeing regular public uprisings about a whole range of questionable privacy practices based in technology and that will only get worse. The fact that teenagers say and do dumb things on Facebook is no guarantee that they'll continue to be that oblivious to sharing all the details of their lives as adults with jobs, families, and assets to protect.

Obviously, all of this is speculative. But then everything about mobile is speculative, a combination of powerful technology and wishful thinking. That's part of what made this conference so interesting and the main reason I was glad I went.

Hats off to Leonard Murphy who seemed to be the prime mover behind the event and whose breathless enthusiasm for mobile picked us up whenever our spirits flagged. If you want to hear why mobile is the future and the future is now, Lenny is your man.

Is there a pony in there somewhere?

Last Thursday I was a panelist in an online discussion about mobile. Jeffrey Henning gave a nice balanced summary here. Leonard Murphy, who organized the event, took a more fanciful and argumentative approach in his report on the Green Book Blog. For those of you who missed it and have an hour to burn there is a full recoding here.

While the panel was supposed to be about mobile the discussion eventually devolved into one of those speculative meanderings about "the next big thing." These discussions inevitably go to all the wonderfully innovative things people are doing and how exciting it is to be alive and in Market Research. In Leonard's blog post, he wanders even further down this road and talks about "the new engagement models" that will get people talking to us again.

There clearly is a lot going on, lots of new ideas and approaches being pitched and new companies that build their businesses around them. The hype being generated is at historic levels, at times almost too much to bear. But is there a pony in there somewhere? If we start shoveling will we find "the next big thing?" I don't think so.

That fact is that despite all of the hype around these new approaches they are not penetrating the marketplace and clients are not embracing them on a broad scale. In terms of spend, they are bug dust in an $8.5 billion US market research industry. It's hard to even see them called out in industry reporting. ESOMAR tells us that as of 2010 online qual, something that's been around for over a decade, rounds up to 1% of total global spend. In the US, it rounds down to zero. I saw some numbers a while back (Inside Research , I think) putting the US spend on MROC's at around $100 million of that $8.5 billion. If anyone has numbers on mobile, I'd love to see them. Neuroscience? Crowdsourcing? Predictive markets? Social media monitoring?

The obvious exception is what some call passive research or behavioral tracking. There is serious spend there and it's increasing. The bet is that we can collect enough data and develop powerful enough modeling so that we don't have to actually talk to people any more. This brings with it a huge issue of privacy which I'll save for another time.

But if you believe that there are a whole range of insights that you can only get from talking and listening to people this industry has a difficult future. The most likely scenario is one in which there are a whole range of methods we will need to choose from, an order of magnitude more than the simple quant/qual world of choices we grew up in. That does not play to our strengths. That, too, I will leave for another time.

Gamification

I spent much of the last couple of weeks heads down with colleagues finishing up a paper on survey gamification for the ESOMAR Congress. We wanted to test whether the claims made for gamification are born out empirically. Betty Adamou has done a pretty good job of succinctly summarizing the arguments and methods here but it mostly seems to come down to respondent engagement.

The idea of gamification in research seems to mean different things to different people. From what I've seen there are three principal applications:

First, incorporating game elements—a story, a challenge, animation, rewards, etc.-- into surveys. Maybe the game is integral to the survey topic and maybe not. In our case it wasn't and we found few of the benefits that others have claimed. (For more on that you will have to come to Amsterdam.) 

The second involves changing the survey altogether and posing questions in more game-like ways. Jon Puleston has been an articulate spokesman for this approach.  In one of his examples he suggests that instead of asking "What is your favorite food?" you ask the respondent to imagine that s/he is on death row to be executed the next day and to describe what s/he wants as a last meal. I agree that many people might find that a much more interesting question but it also is measuring a fundamentally different construct. This approach would seem to require a lot of careful thought and pretesting to understand exactly what's being measured and how it relates to the client's business problem.

The third application is use of problem-solving games in qual groups. I've not seen much about it but it strikes me as the most intriguing of the three, although I confess that I'm not a quali.