The Survey Geek

The Online Evolution 2008 Continued

OK, back online with The Online Evolution 2008.

1:30PM – This is to be an interactive session led by Nick Buckley who is directing the social media initiative at GfK in the UK. The general topic is supposed to be about understanding Online Evolution with a bunch of subtopics. My table has been assigned the topic: Effective Conversations. I won't go through all of the details, but here are the takeaways:

• Communities might be an excellent tool for knowledge management, especially tacit knowledge. Key is that these cannot be imposed by management but need to be somewhat spontaneous among people with common interests.
• It's interesting that a couple of people have successfully used LinkedIn to do this.
• Employees mostly think they are cool but management worries about them being time wasters.

3:00PM – Now for "more research oriented." What we see is not only a lot of traditional advertising being done in communities and so we also are bringing over traditional methods (panels and surveys) to the community space. Does this make any sense?

The "power law of participation" shows that lots of people are willing to click and read but only a few are willing to really interact and write. With time we see more and more growth in writers. Game mechanics reflect the basics of communities as well: collect things, earn points, be able to customize stuff, get and give feedback, and exchange stuff. These are the key principles of computer gaming and also of YouTube.

Ok, finally a research app. Product or brand focused-communities provide an easy way to measure the response to new product introductions. A social bookmarking site like Delicious can provide a lot of insight into what kinds of people prefer what kinds of brands by showing their other interests..

3:30PM -- A panel has now formed specifically focused on how the research business can play in the social media space but it seems to have immediately slipped back into the current world of panels, online research, etc. rather than new media, communities, etc. The big question is what is the likely impact of the financial crisis on MR? Since there is no consensus I'll just tell you what I think. When there is a financial crisis in the stock market there is a flight to quality. In research, clients run away from quality. With the advent of online we started down a slippery slope on continually redefining what is "good enough." With communities and mobile we are broadening that definition. Hopefully the counterbalance is that there is so much other information—survey and non-survey--on topics, other points of measurement, that good interpretation is possible. In other words, translating those "good enough" survey results into real insights by calibrating it against what else we know.

With that bit of wisdom I am signing off. It's been fascinating. I find it very hard to get this kind of discussion in the US, a problem we ought to solve. But right now I'm off to sign up for Facebook. After all, the fasting growing demographic on Facebook is the older 40 crowd!

ESOMAR Panel Research 2008: Day One

This post is a bit late, for which I apologize.

I'm here in Dublin at the Panels Conference.  We began the first day by hearing the Program Chairman declare that this was the last panels conference.  By that he meant that the next one (if you can draw 200 people like this one ESOMAR will keep having it) will likely have a different focus--communities, Web 2.0, Web 3.0, social media, etc.  It seems that panels are passé. Or, to put it another way, panels are paradigm 1 and all of this other cool stuff is paradigm 2.

This was followed by the keynoter, Lorenz Bogaert, founder and CEO of the European social networking site, Netlog a European version of Facebook.. His story is all about another of these social networking sites that have grown up and now have millions of people who sign up, post, and interact with others on all sorts of topics and in all sorts of different ways. Where the principal medium used to be email, it's now blogs, and video postings, IM and even SMS. The whole question of how we use virtual gathering places to do research was left to another day.

The next speaker: me. I felt like John McCain following Barack Obama. This old gray haired guy still talking about probability sampling (paradigm 0, I guess) and how clients can use paradigm 1 to generate great business insights. Or, the importance of not viewing these new methods as replacement methodologies but rather as complimentary methodologies that each tell us different thing, maybe even using them in combination to tell a full story to clients. Desperately trying to stay relevant, although old friends were very kind with their comments at the lunch break. "Pathetic" is the word that comes to mind.

Then three successive papers on client communities. Lots of nice pictures of Web sites, stories about how much participants love the interaction, and testimonials to business uses. The clients who run these communities do indeed float concepts and ideas in front of members and make decisions about new products and service offerings based on the feedback. While in general the presenters understood the compromises they are making the intensity of the feedback and the speed with which they can get it tips the scales in favor of the approach. Perhaps the most interesting aspect of all of this is the intimacy that gets created between the "research" supplier and the client. Each presentation featured both the supplier and a client and throughout the conference the each of these pairs has been joined at the hip. It's clear that they interact constantly with one another and it creates a bond and synergy that is envious.

One observation. When pressed, smart researchers will admit that panel research is not really as accurate as traditional probability-based methods but the speed and cost benefits trump that to a point where we are happy to accept "good enough" research. (How is this like or unlike "quick and dirty?") Communities may be the next stop down on this slippery slope.

Bottom line: interesting but not especially illuminating.

The afternoon started with a client panel. These things seem to flop more often than they work. The clients are all drawn from conference attendees who in general are folks in the trenches rather than more senior people who deal with broad strategy or policy making. So the discussion is rather mundane. About things like whether to use a Back button in surveys.

The next group of papers was more promising: two on conversion of ongoing studies from offline to online and one on mobile. (Sadly, this is about the point when North American began to wake up and my email because a significant distraction.) The two on conversions were mostly disappointing stories about attempts to convert from offline to online and having all sorts of problems producing convincing numbers. One paper title, "Turning the super tanker," went into a good deal of detail about the infrastructure challenges as well as data quality issues which while interesting struck me as things I heard before. I'm not sure what to make of this but in the place where you are supposed to take notes in the program booklet I wrote, "Not very interesting." At the conference closing it was nominated as one of the two best papers at the conference.

The best paper of this lot was by Otto Heilwig from Respondi in German. Frankly, I found his presentation to be a bit difficult to follow and I must read his paper more carefully. He seems to have done a pretty well designed experiment that compared use of mobile phones to collect quick reactions to events with use of traditional telephone. I don't his results from the experiment were all that interesting but the paper did a nice job of pointing out the main challenge to use of mobile phones as a data collection platform. Chief among them is the difficulty of surveys with SMS, and while something close to two-thirds of German mobile phone users have Web-enable phones less than 15 percent use them to access the Internet because it's just too expensive. During discussion at the evening reception many of us agreed that given current limitations with the devices probably the next phase for mobile will be more around replacing and improving on IVR than on replacing "traditional" online.

So a somewhat interesting day with some food for thought but nothing extraordinary.

The Online Evolution 2008

I am at the ESOMAR World Research Forum: Online Evolution. The last two days I have been at the ESOMAR Panels Conference in Dublin. I have meant to blog on that each day but it's proven to be very difficult to find the time. It's the sort of conference with little downtime and my posts are still stuck in word. Today I am going to try doing this in real time. This is my first experience in "live blogging."

9:00AM – This conference is mostly client-side, "business people." It has about 50 attendees. Right now the president of ESOMAR is talking about the Celtic Tiger which is the story of Ireland's rise from the poorest country in the EU when it joined 25 years ago to now its richest. (This is the same intro he gave to the Panels Conference). I am sitting next to and sharing the power connection of a reporter from MR Web and he is typing a lot faster than I am.

9:05 AM – Marc Drüner, CEO of trommsdorff+drüner, is doing and overview of the program and now transitioning to a discussion of Web 2.0, social networks, communities, etc. In other words, picking up where the Panels Conference left off. When we think of communities we think of things like Facebook but there also are special interest communities that are growing up like weeds in a garden. What all of these have in common are the constant streams of communication and updating of information. More and more people on the Web communicate less with email and more with exchanges within and among communities. The challenge for marketers is to figure out how to get their messages into these communication streams. Classical marketing was all about banners and ads, but that doesn't work for communities. Click through rates on Facebook are miniscule. People are in a social mode, not a buying mode. Marketers need to get into the conversation not try to interrupt it. The master at this has been Barack Obama who has set all kinds of records with 2.2 million friends on Facebook.

Now he's talking about "the Googlesation of the World." So not only do we have things like Chrome, the Android operating system, and software as a service, but the digitalization of all print/content. And now they are moving into TV with access to set-top boxes that make it possible to sell advertising to "long tail" companies. Now for the scary part. Imagine you are watching TV with a set top box that Google controls. A friend calls you on your iPhone or you are using your laptop, surfing the Web with Chrome. Google now knows exactly what you are doing at every second.

I'm not sure (yet) what this has to do with MR but it is fascinating!

9:35AM – Next speaker is Susan Wegner from Deutsche Telekom. She is talking about the evolution of mobile phones/devices. The story is not just about increasing penetration but the expanding functionality and migration of applications from PCs to mobile phones. She is doing a somewhat technical overview of the software architectures of PCs and phones and the gradual drift toward consistency between the two so that users have a common experience regardless of platform. There are a number of challenges. One is for providers to continually expand the bandwidth needed to support it. Another is improvements in the networks so that services can be delivered consistently no matter where the user is. In other words, an end to "dead zones." It's all somewhat technical but a good sense of where mobile is headed.

Now she is shifting from retrieving content to inputting content. She notes that mobile phones make it tough to input because the keyboards don't work as well as they do on PCs. So there need to be other ways to input content—cameras, text scanners with OCR capabilities, sound files, tools that help us to snip content from online sources, etc. GPS creates a location for a user and possibly even context that can control the delivery of information in a very focused way.

9:55AM – Unfortunately, the third speaker in this session had air travel challenges and so he is not here. Here topic: "Web 2.0's influence on social change." I would have loved to hear that.

So we are now wrapping up the first session whose general topic was, "The Online Evolution."

10:00 – Refreshment break and I am feeling very smart. In 1996 I wrote:

In 1987, to commemorate its 50^th anniversary, The Public Opinion Quarterly asked 16 well-known scholars and survey practitioners to offer their visions of the future of public opinion research (Bogart, 1987). In his response, Harold Mendelsohn described how ". . . the eventual 'computerizing' of the American home undoubtedly will contribute significantly to the speed, accuracy, and economy with which data will be gathered, analyzed, and readied for dissemination." James Beniger wrote that "a host of new technologies will . . . make possible the real-time mass monitoring of individual behavior . . . Survey research will increasingly give way to more direct measures of behavior made possible by new computer-based technologies." In a particularly chilling vision, and perhaps somewhat tongue-in-cheek, Robert Worcester described how ". . . market researchers are close to their tactical ideal, the comprehensively wired micro model of segmented household 'norms' which can be conceptualized, pressurized, test marketed to, weighted (up and down), copy tested, product tested, studied, and, yes, manipulated by cables, satellites, and sensors (worn in rings, necklaces, earrings, or even—shades of George Orwell—implanted!)."

The foundation for these visions of the future is one that is increasingly popular among futurists such as Nicholas Negraponte, Don Tapscott, Michael Dertouzos (1997), Gregory J.E. Rawlins (1997), and William J. Mitchell (1995). It has as its beginnings in Norbert Weiner's seminal book, Cybernetics (1948). The vision is one of a technological utopia, a vision of a completely digital world in which everything is smart--home appliances, automobiles, clothes, money, buildings, highways. In this utopia, computers will be omnipresent, and everything we use and touch as we go about our daily lives will be busy collecting, processing, and transmitting information. The implications for survey research are extraordinary.

I admit that when I wrote it I believed it but had trouble really imagining how it would happen. Now it's clear. I went on to try to explain those extraordinary implications for survey research but in rereading that I don't feel so smart.

10:30AM – The chair is now introducing the next section which is titled, "Business Applications." The first speaker is Keith Bailey from Nokia Consumer and Customer Insights. His topic is: "The Future of Work." His job is all about developing mobile products for business users. They have been doing research on the needs of workers at various levels inside an organization. Some societal drives that need to be considered include the changing demographics of work (diversity and generational change), globalization, environmental engagement (energy costs, reduced travel and greening of the workplace), growth of the knowledge economy, the interplay of human instincts and business needs, and the technological watershed we are now experiencing. Now he has moved on to emerging business models, a key one of which is new methods of collaboration that includes virtual teams, nomad workers (sort of like me right now), work-life balance, and knowledge management. Interesting stuff but not clear how to leverage for research. That's promised for this afternoon.

11:00AM – Next speaker is Patrick Eikelenboom, Director of Digital Marketing at Mars in Germany. His topic is fostering best WOM practices in FMCG (a.k.a CPG in North American). Even though Mars is best known for its candies he is going to talk about—are you ready—pet foods. Worse yet, for cats. I feel like I am experiencing a new low. My darkest fear when I came into MR. We'll see. He starts from the premise that communication is changing. It used to be that only mass communication (like TV) had high reach, but more and more one-to-one communication has high reach as well. Basically the whole talk seems to be about how to exploit social media to get out their marketing messages. A lot of cute tricks but nothing very impressive so far. And it never really got all that interesting. I don't think WOM every really came up.

11:25AM –Next up is Ted Speroni from HP/France who is going to talk about building online B2B communities. He runs the European version of hp.com. Initial thrust seems to be setting up Web sites that encourage people to print stuff. Like a site where you can access project software that helps you to make banners, greeting cards, etc. They also have tried various sorts of YouTube related things to promote their products. But the main thrust of this prez has been to set up IT support sites on use of their products. It's a place where people can go and ask questions. In the beginning, experts answer the questions but the goal is to migrate to where visitors answer one another's questions. They seem to have reached critical mass where there is enough traffic and activity to sustain itself. They harvest the commentary for marketing messages back to the IT community.

11:45AM – The last speaker of the morning is Malte Friedrich-Freksa by YOC, Germany. He is going to talk about using mobile technology in the context of digital market research. He describes himself as a market researcher for a marketing company focused on mobile. They tried some research by SMS in 2003 and it flopped. Next step was to develop a panel. The model was downloading of an application through which panel members could report on shopping. It flopped, too. Now they are focused on exploiting the mobile Web for research. Now right away I am remembering that yesterday at the Panels conference I learned that while about 60 percent of Germans have Internet access from their mobile phones only about 13 percent actually do it because it's expensive, a bit slow, and browsing can be clumsy. Despite that they continue to explore feasibility. Their initial work has focused on understanding whether people will participate in this kind of work, why, and to understand their experience. They are using standard Web survey software from Globalpark. Results were not encouraging. Only about 16 percent of their panel completed the survey. The single biggest reason for not doing it was the cost of mobile Internet access. They also asked people whether they preferred online or mobile surveys and online was heavily favored, although for short surveys (6-7 minutes) about a third said they would prefer the mobile platform.

So that's the morning. Now for lunch.

Two new books worth a look or maybe owning

In the last month or so two good books on Web survey design have appeared on the scene.

The first is Designing Effective Web Surveys by Mick Couper.  In the interest of full disclosure I admit that Mick is a good and long time friend with whom we have done a lot of work over the last five years, but I urge anyone who works with Web surveys to get hold of this book because it's a terrific overview of the key elements of Web survey design.  There may be a few things here and there that I disagree with, but Mick pretty much always has the data on his side.  But as Mick warns in the preface, it's not a cookbook.  It was, after all, Mick who taught me that the answer to every methodological question is, "It depends."

The second book is a follow-up of sorts to an earlier book by Don Dillman.  This time he has teamed up with two of his graduate students to produce a third edition of Internet, Mail, and Mixed Mode Surveys.  For those of you who don't know Don, he is THE mail survey guru who virtually invented how mail surveys are done.  So it was something of a natural for him to move from one self-administered mode to another.  Along the way he has done a lot of good work on questionnaire design and recently has become very interested in mixed mode.  I've not seen the book but knowing the people involved it's probably worth a very close look.

Sadly, both of these books are pricey. The Couper book is $125 in hardcover,although a paperback is available.

Engaged or still bored?

I have just returned from THE Market Research Event. Yes, that's how they promote it. I was only there a single day due to some heavy scheduling conflicts but in my limited time I heard a lot about "respondent engagement." Everybody is indeed talking about it.

Socratic Technologies has been doing a lot of this stuff and in purely technical terms they are doing as good a job at it as anyone. Their demos are worth a look. At the conference they showed some pretty impressive applications with claims of improved focus and better data based in the psychology literature although not much data to back it up.

As readers know, I am a skeptic on this because I think the real problem is not look and feel (in most cases) it's just surveys that are too long and too boring. As a conference attendee I was sent a link to a survey that had some cool flash built into it. On the screen below I was asked to use my mouse to sort the list of conference features into one of the five buckets based on its attractiveness. The execution was very well done, although preceding screens testing for the right version of Flash and then forcing me to watch a demo of how to do the task made me a bit impatient.

When the time came I charged in, but after I had done what seemed like 20 of these features I began to wonder how long this would go on. As you can see by the screenshot, once I looked the news was all bad. First I noticed that I still had 32 attributes to go! Then I noticed that it threatened another exercise after this one. I missed it in the screen shot, but there was a progress meter in the lower left corner telling me that I was 19 percent complete. So for me, at least, no matter that this was a cool way to answer. The survey was just too uninteresting to me and too long for me to continue. So I bailed out.

I suspect that as time goes by we will see more and more of this kind of eye candy and we probably will do it ourselves because clients will want it. But I just don't think it solves the real problem here which is that most surveys are too long and too uninteresting for many respondents. What they really want are shorter surveys on salient topics with well written questions. It turns out, that's a lot harder than you might think.