The Survey Geek

Whither mobile?

I spent two days last week in Chicago at Market Research in the Mobile World (MRMW). I now have been to all four of the North American versions of this event, all the way back to Atlanta in 2011. Last week’s iteration caused me to go back and look at my post about the Atlanta event as a way to gauge how much things may or may not have changed.

One obvious change was the sorting out of marketing applications from genuine research. In Atlanta we heard from too many companies whose business was collecting personal data for direct marketing, and not always transparently. As far as I could tell, the presenters in Chicago were primarily focused on research. A second change was the emphasis on case studies. Atlanta was mostly about the potential for mobile—what could be—while Chicago was mostly actual studies completed.

Atlanta also was heavy on hype and there was plenty of that over the two days in Chicago, including a fair amount on the potential of wearbles. But it was hard to get too worked up about any of it given the sobering start.

• The first presentation was a paen to mobile as “the most important marketing channel, ever” that included the claim that 55% of people really like targeted ads, a figure undermined by virtually every credible survey on online privacy.
• That was followed by an update on last year’s eye-popping announcement by General Mills that they hoped to be doing 80% of their research via mobile in 2014. No exact figure was given but it was clear that it turned out to be harder than they thought. They have dismantled their mobile team,  but they keep on plugging.
• The segment was closed out with a panel of industry heavyweights on the topic, “Investments in (mobile) MR—where are they going and why?” The answer: they are not. The money is all going to big data analytics.

That was followed by about 30 presentations with an overwhelming emphasis on pure mobile applications: in-the-moment, geolocation, and mobile ethnography (as opposed to the unintended mobiles that comprise the bulk of mobile MR right now). Some were genuinely interesting and others seemed like sales pitches. The highlights and lowlights in no particular order:

• There were a couple of nice papers on the topic of integrating mobile with other platforms as a way to understand the context in which people view different ads.
• I heard no discussion of the sampling challenges beyond a preliminary report from TNS aimed at allaying concerns about bias in online surveys that include respondents using mobile devices. One might have gotten the impression from most presentations that there is 100% smartphone penetration and people are willing to use them to do pretty much anything researchers ask them to do.
• The potential power of mobile ethnography was nicely demonstrated by several presentations.
• There was a somewhat bizarre though seemingly heartfelt epilogue to one presentation pleading with us not to be swayed by the media into giving up on Google Glass.
• There was an equally bizarre presentation on mobile’s ability to reduce social desirability bias and satisficing that opened with the presenter acknowledging that all he knew about the phenomena was what he learned from their Wikipedia entries. It seemed to be just another study on recall.
• There continues to be some sloppiness around proper privacy protections, especially in mobile ethnography. There was a panel on the topic (I was on it) but less than half of the attendees were in the room.
• A number of presentations took up the theme of respondent-driven design that emphasized better interfaces, choice of channel, and better-designed if not shorter surveys. On this latter topic, one presenter showed that a well-designed 15-minute survey is possible with no discernible data quality drop off.  No doubt music to the ears of clients (of which there were very few) and even researchers who cling to long surveys like an NRA member clings to his assault rifle.

My bottom line is that things have changed substantially since Atlanta, but nowhere near as much as mobile evangelists predicted. Mobile has become mostly an extension of online, and while there is no shortage of startups offering in-the-moment and other pure mobile solutions there also are no clear signs yet of research buyers flocking to them. Mobile has changed substantially how we interact with one another and the world around us. It must be a fundamental concern in every research design, but it has yet to truly transform MR in any meaningful way.

A little theory might help

As best I can remember it was November of 2009 when I first sat through a conference presentation setting out some basic design considerations for mobile research. I can’t imagine how many times since someone has presented essentially the same advice whether from the conference stage, in a corporate white paper, via a webinar, or whatever. Mostly these rules, guidelines, tips, etc. are based on some combination of experience and intuition. They generally lack any theoretical basis or methodological experiments to back them up. That’s for the academics. Market researchers just want to get on with it, just do mobile. It’s the same approach we took with online surveys and perhaps one of the reasons why they often are so wretched.

Having some theoretical underpinnings for mobile design strikes me as a good thing. It might provide a rationale for why we do what we do, and serve as the as the basis for hypotheses to drive empirical experiments that might actually lead to better practices. Perhaps more importantly, a good theory could be really useful when helping clients to rethink their approach to surveys in a mobile world.

All of this came to mind while reading this post by Raluca Budiu arguing that information theory has a lot to offer as we think about designing for mobile devices. In a nutshell, her argument is that designing for mobile is all about dealing with basic limitations in the human-device communication channel. Some of those limitations are technology-based, such as the bandwidth of the connection to the Internet, the processing capability of the mobile device, and the size of the screen. Other limitations have to do with the user, things like working memory capacity and the focus that a user can bring to the task given the environment. Granted, Budiu’s principal focus is on websites, not surveys, but it is even more important to offer an optimal design for surveys because the motivation generally is much poorer when completing a survey than when searching for information.

And, as Budiu points out, things are about the get a lot more complicated.

It’s clear that we’re moving towards an interconnected world populated by a plethora of devices — from smart thermostats, smartwatches and smart glasses, smart phones, phablets, tablets, laptops, desktops, smart TVs, and smart tabletops. We need a unified theory for designing for the continuum of screen sizes. This theory cannot reduce all these systems to a single denominator; designing for smartwatches is not the same as designing for tablets, and designing for mobile is not the same as designing for the desktop. Although many of the principles may be the same, they get applied differently on different devices. We need more nuance.

The challenge may be even greater for researchers than for website designers. One of our goals is ensure that the way in which we present a survey does not cause the respondent to answer differently because of the device he or she is using. Having a little theory to guide us could be a big help.

 

If You Need to Know What’s Legal, You’re Already on the Losing Path

This is the second and final post from Michael Link on location.

Privacy and ethics concerns in research are not new, but they have taken on considerably higher visibility in our 24/7 news world as researchers test the bounds of new measurement approaches. At a recent symposium on Leveraging Location hosted by Nielsen, a panel of legal experts provided some thoughts in this area on issues researchers are encountering today. Their insights went well beyond location data, hitting on aspects that involve many of us working with data from the public. (Obligatory warning: I’m not a lawyer; these are simply my observations and interpretation of the discussion). Three broad lessons caught my attention:

First and foremost, start with the respondent/consumer, understanding and acting in accordance with their expectations. How data are collected and the insights generated should be readily apparent to the “average person.” If your starting point is “the law” or “what is legal”, you’ve already put yourself in a hole. Laws and regulations provide a base -- a bare minimum, what the public demands is often much more. As researchers we should operate within the reasonable expectations of the majority of the public, yet not necessarily feel constrained by folks on the extremes.

Second, to lead in innovation you cannot be afraid to have your name in the paper and receive negative comments. In essence, as one panelist put it “get comfortable not being comfortable.” Pushing the envelope involves a degree of calculated and real risk. If your organization likes to keep a low profile and acts with alarm at the first half-dozen negative emails received, then you might want to take “innovation leader” off of your long-term business strategy. Note that Rule #1 above is still in effect, so setting expectations accordingly and having a good grasp of the potential risks is imperative.

Third, time is an important and often under-appreciated dimension of attitudes towards privacy and ethics. These are not static features of our society, but rather evolving concepts. What may have been unthinkable a few years ago (using a smartphone as a “virtual wallet”) now seems commonplace. Likewise, certain aspects of data collection in this digital/organic data era that seem unreasonable to the public today (and hence, would be good to avoid) may become more readily accepted with time and incremental exposure. The trick to innovating, of course, is knowing when the time may be right. To this there is no clear right answer, but the public (and the press) will let you know if you have chosen unwisely.  

Storytelling comes to the conference podium

The first two presentations at the conference this morning have helped me to understand what it is that I'm finding so disquieting about this conference. What we are hearing is not the usual kind of research conference presentation. What we are hearing are stories. No data. No experiments. No hypothesis testing. Just stories. Oral blog posts from the podium with some neat visuals by people who are smart and articulate. 

OK, so I probably am exaggerating some but it makes me wonder whether this is where the current emphasis on "storytelling" is taking us as an industry. And, of course, is that a bad thing? Are we really connecting the dots or just making unconnected leaps of logic to get to a view point? In my previous post I worried that we are too focused on "just the facts, ma'am" as a deliverable. We probably should be just as worried about wonderfully-sounding insights that are disconnected from those facts. 

 

Kids in a candy store

I'm at the Market Research in the Mobile World Conference in Cincinnati where the constant stream of cool MR mobile applications is off the charts. You can't help but be impressed with the creativity being brought to the task of using mobile to give us a whole new perspective on consumer behavior. And the people showing their wares here in Cincinnati all seem to be having a great time doing it. The excitement is palpable.

But the most interesting session of the day to my ear was the client panel to which our host and MR impresario Lenny Murphy posed the question: "What's the gap between what clients need and what suppliers are offering?" The answers all seem to boil down to our failure to help them understand how all of this cool new stuff and the data it produces can be put to work in their companies. There were lots of specifics—boring reports that need to be rewritten, failing to help internal stakeholders integrate the data into their existing databases, not being clear about how new methodologies do what they purport to do, not understanding their business and industry, etc. It's certainly not the first time we've heard this from clients. And I've seen zero evidence so far that mobile helps.

Clients have been complaining for years that researchers mostly just deliver facts, or what they claim to be facts, and making real business sense of those facts is not a deliverable. For all the talk about the importance of storytelling, it's just another way to present the facts. Ditto for most of the cool video and animated deliverables being generated out of mobile. Clients want us to connect the dots from those facts to their business and they keep telling us over and over that we're not getting it done. I guess we're just having too much fun with data collection.