Previous month:
December 2011
Next month:
February 2012

Posts from January 2012

Plus ça change – Part 2

Back in December I wrote the first of two posts noting the way in which some trends in our industry can rise and fall and then rise up again with all of the same promise, problems and fears. I noted the way in which the issues surround CRM and data mining seem to be having a second act in the big data movement. This is the second post and it's about the way in which the industry's drive to improve online questionnaires, knowingly or not, is drawing on some longstanding principles that until now have fallen out of favor.

One thing that most people in this industry seem to agree on is the sorry state of questionnaire design. We have redefined "good design" in terms of the things we can measure. So we count up the number of grids, page loads, words on the screen, etc. And at the end of surveys we ask the survivors how the experience they just plowed through compared to the vague benchmark of "other online surveys you have taken. " Back in the salad days of survey research we used the metaphor of conversation to describe a survey. The operative metaphor today seems to be closer to root canal.

But I see signs of hope. The first is what Jeffrey Henning has been blogging about under the heading of "Crowd-Shaped Surveys." The principle is simple: if you want questionnaires that work better for respondents then involve them in the design process. ThinkAloudThis principle is also a very old one. For decades researchers have used techniques such as focus groups, cognitive interviewing, think-aloud exercises, and pretests to fine tune survey questions so that they are easier for people to answer and yield the information that the researcher is looking for. These techniques are still widely practiced in academic and government circles but seldom used in MR. I guess we think that we just know how to write great questions. Jeffrey has been sharing a lot of ideas about how we can achieve some of the same goals via technology. I'm not crazy about all of what he suggests and look forward to seeing the validation studies, but it is one very promising path out of the rut we seem to be stuck in.

The second sign of hope is survey gamification, or at least some elements of how it is practiced by survey gamification guru Jon Puleston. (Jon also is an advocate of pretesting.) Now I admit that I'm no fan of Flash gadgets or many of the gaming features Jon has tested, but as I have written elsewhere one key element of Jon's approach is simply to write better questions by making them specific, creating a context, giving examples to clarify and trigger recall, etc. He reads the literature on questionnaire design and it shows in his work.

At the end of the day good questionnaire design is not about grids, page loads, interactivity, cute graphics or fancy answering devices. It's about using words well to ask questions that are clear and interesting to respondents and to provide a meaningful set of response options to capture the answers. That can be a tall order given some of the topics clients ask us to research, but there are ways to get there. It's encouraging to see MR remembering them.

ISO 20252 – Market, opinion and social research

In December we were audited for ISO compliance. After a week of staff interviews, project reviews and records inspections we were certified to the 20252 standard. Iso Good for us!

Now I recognize that there are more than few people in the industry who think ISO certification is some kind of retro thing that does not reflect how much the industry is changing. More on that in a bit. But first let me say that when it's all said and done ISO 20252 comes down to four pretty basic things:

  1. Making sure that people know how to do their jobs and can succeed in the assignments we give them.
  2. Ensuring that everyone on a project team understands their individual roles, the roles of their colleagues and how to work as a team to produce great work for their client.
  3. Being completely transparent with clients about what was done and how it was done so that they can make whatever business decisions they need to make with confidence.
  4. Minimizing the mistakes that generate rework and disappoint clients while maximizing the chances of learning from those mistakes.

Now back to those folks who say that ISO is for the dinosaurs and out of place in the fast-moving world of contemporary market research. Or that all that process ISO insists on works against the innovation that is the supposed life blood of boutiques and startups. I think that's nonsense. A well-defined infrastructure of process and procedures need not be a barrier to change. In fact, it can make it easier. In Good to Great Jim Collins observes that in great companies people work within the framework of a highly developed system that fosters what he calls "a culture of discipline." When you have a culture of discipline you spend less time managing people and more time managing the system. And when change is called for you change the system, not the people. That's also what ISO brings you. I recommend it to you one and all.

Is mobile finally ready for takeoff?

Not that I claim to have read them all, but the one thing that most of the year end MR prognosticators seem to agree on is that 2012 will be the year in which mobile finally takes off. But will that takeoff be more reminiscent of the Wright brothers or the Saturn V?

Larry Gold shows us some interesting numbers on this topic in the January issue of Inside Research. For a number of years now Larry has been maintaining an index of online spend using a group of 25 Honomichl Top 50 companies. (Here I note that my company is one of them.) These numbers are probably the best we have for charting the growth of online over about the last 15 years. On the last round Larry asked participating companies to report revenues for mobile data collection. Wright-brothers-flight Looking at a three year trend he estimates for US only $9.4M in 2010, $20.1M in 2011 and a forecast of $32.9M for 2012. That's an increase of 114% in 2011 and 64% for 2012. Impressive growth for sure! And as he points out, he misses revenues from smaller companies who are mostly focused on mobile, although it's not clear how much those smaller companies are likely to increase the overall number.

A couple of comments. The first is that in terms of spend mobile is now about where online was in the late 1990s (without adjusting for inflation) when revenues were doubling and tripling every year. It took until 2004 to top $1B and it's now forecast at $2.5B for 2012. So despite its recent growth, mobile is still in its infancy. The second is that mobile may face stronger headwinds than online. Rightly or wrongly, a lot of the conversion to online from other methods was pretty simple: just administer the same questionnaire using a different sample source. SatvdOnline also came with a killer app in concept and product testing. Moving to mobile is a good deal more challenging because most of those questionnaires are going to need massive reworking before they are mobile ready. There are some interesting things being tested in terms of breaking up questionnaires into smaller chunks but that's still mostly in the testing stages. And mobile's likely killer app, "in-the-moment research," may not be enough to make it as dominant as online has become.

I don't question that mobile is poised for significant growth. But how soon, how fast and how big it will get are still open questions.

The (wireless substitution) beat goes on

To no one's surprise the gold standard on wireless substitution in the US, the NHIS, reports that the proportion of wireless only households keeps growing.  As of July, 2011, it was 31.6%.  That's a 1.9% increase since December of 2010 and translates into 30.2% of all US adults.  Just eyeballing the graph below it does not appear that the slope of the line that matters the most to us--the blue line--is changing much.  Hard to believe there are still people out there doing telephone research without calling cell phones.