The Survey Geek

No surprise: It just keeps getting worse

The latest numbers on wireless only households as measured by the NHIS have just been released.  They now estimate that as of December 2008 20.2 percent of US households have only a wireless telephone.  The 2.7 percent increase in the second half of 2008 is the largest six month increase since the NHIS has been collecting these data. 

I recently saw some estimates from SNL Financial that appear to have taken the NHIS trends and projected them forward.  Their data say that by 2013 a whopping 37 percent of US households will be wireless only. 

As I have noted in the past, the most statistically robust way of dealing with this problem is to spend the extra money and dial some cell phones.  The financial hit is not too bad as long as you only have to do limited dialing.  But as the proportion of cell only Rs increases, so do the costs.  While there is always the option of mixing in wireless only Rs from an online panel that is a journey into a statistical no man's land.

New report on wireless substitution

Kate Harris has pointed out to me that CDC  just released a new report on the prevalence of wireless only households in the US, and this time they are providing state level estimates.  The variation across states is dramatic to say the least.  Oklahoma is estimated to have a whopping 26.2 percent of its households unreachable via a landline phone while Vermont is at just 5.1 percent. 

These estimates are based on 2007 survey data when the national estimate was 14.7 percent of households.  In 2008 that estimate rose to 17.5 percent.

Mobile Research Conference 09 - Day 2

I’m sure you’ve noticed that 3 weeks have elapsed and I’m only day 2. Well, a few things intervened. Though I’d sum up the rest of the conference – which ran the gamut from strong presentations to a fascinating but somewhat tangential soliloquy on response rates – in very simple terms. What did I learn you should and shouldn’t do in mobile research?

Should

1.       Use on immediate, emotive or recall, and location-specific topics. Examples of ideal mobile survey question: How attractive is that package you just picked up at Tesco? Which advertisement do you remember from the commercial break that just ended and did you like or dislike it? Ipsos found that ad recall was more accurate in mobile vs. web surveys though verbatim were much less rich.

2.       Expect to get nearly all of your responses very quickly and if you don’t want that, stagger release of the sample.

3.       Recognize that unlike other modes, it costs respondents to participate (not just their time), even down to a per-response cost if you use SMS.

4.       Consider appropriate incentive structure that compensates for cost of participating (SMS or data plan costs) over and above the typical remuneration for time/trouble.

5.       If working with a panel provider, determine whether their mobile panel is purpose-built or simply regular panelists who opted in to mobile surveys.

6.       Use an enhanced invitation (see Ipsos paper for example): personalized, emphasizes that participation is free, states purpose of study (Total Design stuff). This resulted in higher response rate and more meaningful verbatims (though it had no effect in an email invitation to a web survey run in parallel, and a subsequent GlobalPark study contradicted the findings on rich verbatim).

7.       Think about how a mobile survey could integrate or converge with a social network so that you can combine a conversation or co-creativity exercise (network) with brief, point-in-measures (mobile survey).

Should not

1.       Use gratuitously. There is no firm evidence that mobile respondents will be more engaged than in other modes/panel types, or that overall data quality or time-to-insights are improved. Examples of really successful implementation are few and far between.

2.       Expect to get older people or broadly representative consumer samples out of mobile panels. That being said, every mobile panel vendor said his or her panel is underutilized at this point so there is room for experimentation.

3.       Give people mobile survey as one among many survey mode options. It is fundamentally different from other modes, confuses people and lowers response rate to have too many options.

4.       Field SMS surveys (as opposed to web-based mobile surveys) if you can avoid them, though the invitation trigger may be an SMS. They are easier and reach a broader base of respondents, but trigger major concerns about cost and are tough to integrate on the back-end since each response*respondent combination is a separate record.

5.       Tell people it won’t cost them to participate once and expect they’ll remember that. In experiments and field results shown, cost to participate was a major concern no matter how often respondents were reminded that it was free. In the Ipsos study, even though SMS was free, half of those dissatisfied with the experience focused on cost.

6.       Expect response rates from mobile panels to be significantly higher than from “normal” internet access panels. Cited response rates were mostly in the 10% to 15% range.

Mobile Research Conference 09

Reg has kindly allowed me to act as a guest blogger. I'm in London at the (first?) Mobile Research Conference, put together by Global Park. About 75 people in attendance, mostly suppliers and academics. Today was the first day and we had 6 sessions ranging from broad keynotes to case studies of specific research projects conducted on mobile platforms. My broad observations at this point are more about what isn't being said than what is:

1. A lot of emphasis is placed on the advantages of mobile platforms as "personal" in a way that web surveys on computers are not. Not honestly sure I understand this. I understand that people have a different relationship with their phones than with their PCs, and that PCs are often shared, but that doesn't mean computers aren't personal, and in any case I'm not sure what practical advantage a "personal" device has for research. I certainly see some of the practical disadvantages – principally that if a research invitation is greeted as spam, it's likely to raise more ire on a phone than in email on a computer.
   
2. Research using SMS and web surveys on a phone browser are the first modes I'm aware of that have cost implications to participants (other than the value of their time). Six questions posed by SMS can cost the participant upwards of $2. In countries with high rates of pay-as-you-go mobile plans, the cost can be quite high. This has clear implications for how we provide incentives, but I suspect it has other implications as well, which haven't been thought through well.
   
3. There is a huge need for sample sources. There is no frame of mobile phone numbers and no real equivalent of internet access panels as a driver of mobile phone research growth yet. The panels that exist are subset from existing internet access panels so its coverage error compounded with coverage error. The panel providers who spoke typically saw 30% uptake among existing panels to participate in a mobile panel, but surprisingly the common remark is that their mobile panels are underused.
   
4. Questionnaire length limitations are and probably always will be a feature or limitation of this mode. We need to seek out more creative ways to use the mode beyond just asking questions. Sweetening responses with other forms of data (location services, photos) helps but it also turns the mode into a vehicle for discussions that are semi-qualitative in nature – very cool but not fitting very well into the model for efficient research that the industry relies on now.
   
5. Every presentation focused inordinately on how quickly people respond. It reminds me of the early panel days when speed was touted as an end rather than a means to an end. I am interested in compressing time to results, but with the exception of certain polling and media research context, I don't think that getting responses in 2 hours rather than 2 days really addresses a client need.

Apologies for any typos. I'm trying to embrace the casual nature of the medium. Those interested in going a step further can consult the conference's tweets at \mrc09.

Increased mobile Web use triggers Web surveys on the move

You have to love the hype. Or maybe not.

The title of this post is taken from an email I got this morning from a company in the UK that, you guessed it, wants to do surveys on Web-enabled phones for us. It goes on to say, " According to a November report by the BBC, there is a notable 25 percent increase in people accessing the web from mobile phones. This compares to a mere 3 percent growth in internet access from desktops. The report predicts that this new wave of mobile access to the web will increase even more in 2009." Does this mean that more people are now accessing the Web from their mobile phones than from desktops? Not exactly. One missing fact: the percent of mobile phone users who use their phone to access the Web.

I am reminded of another bit of news in this morning's email: Internet penetration in China increased by 43 percent last year bringing the current level to 22 percent. Big increases on a small base are misleading.

But back to the mobile story. As I reported back in November there was interesting commentary on this both in a paper at the ESOMAR Panels Conference and at the follow-on Online Forum. The country at issue there was Germany but it's hard to imagine the situation is much different elsewhere. Surveys on mobile phones have three major problems:

1. While more and more phones are Web-enabled, people make limited use of the functionality because it's just too expensive. In Germany, for example, roughly two-thirds of mobile phones are Web-enabled but less than 15 percent of mobile users actually access the Web from their phone.
2. There is a significant bias toward young people, and I mean really young people with a heavy dose of teenagers.
3. The interface is very challenging. Just porting over what we are used to doing on the Web is the road to failure. Questionnaires need to be dramatically shorter and presented much differently on the small screen.

What troubles me most about this is the way in which hype overcomes good judgment. The imperative to innovate with technology in MR has created a frenzy of new applications surrounded by hype and false claims that I think confuse clients and ultimately devalue what we do. I am fond of citing Colm O'Muircheartaigh's definition of error in surveys: "work purporting to do what it does not do." The issue here is not just one of survey error. The issue is basic business ethics.