The Survey Geek

AAPOR catches the wave

I’ve been going to the AAPOR Annual Conference for more years than I can count.  The last several have been like trips back in time.  While web survey design was often discussed the the new and rapidly-evolving methodologies that occupy MR conferences were seldom acknowledged in any meaningful way.  Or, if acknowledged hardly welcomed.   But this conference was dramatically different.  There were five sessions on mobile, another four on various aspects of social media and three sessions on non-probability sampling including a plenary noticeably absent the fireworks that have accompanied similar discussions in years past.  There was even a session on gamification!

I think this is a really important development.  As new methods move front and center with this group of researchers we can expect our understanding of them to deepen and the methods themselves to become more robust.  Kudos to conference chair Dan Merkle for a job well done.

The Mosaic of Global Research

Last week I attended the much-retooled CASRO Global Conference in Miami. The event was co-sponsored by CASRO and ARIA, the Americas Research Industry Alliance. There were around 100 attendees and the program seemed to me to be stronger than in previous years.

Day 1 was interesting. The highlight for me was a presentation by Simon Chadwick on what I would call the business side of MR. He described the evolution of international MR that has culminated in an industry structure in which a very small group of large global companies dominate. Using what Simon called "an imperial strategy" these firms have planted their flags in countries all over the world and, on the face of it at least, are positioned to conduct research anywhere on behalf of the multinationals that comprise the heart of the industry's client base. The driving force behind this strategy has been the emergence of a global/consuming middle class, first in North America, Europe and Japan, then the BRICs and now emerging market countries worldwide. Within 10 years we can expect that 90% of the global middle class will be outside of North America and Europe. The question Simon posed was whether the imperial strategy will work in that future.

He suspects not. Two key developments to be reckoned with are (1) the growth of multinationals outside of the US and Europe and (2) the emergence of new data collection modalities with the global infrastructure to support them. The former is especially important as the percentage of MR work being contracted out of the US and Europe declines while increasing in countries where multinationals like Tata and Lenovo are headquartered. In this context, regional MR firms may be able to compete successfully against the big global companies. And the continued increase of online, mobile, social media, etc. means that having feet on the ground is not as big an advantage as it once was. With the right set of partners it's now possible to conduct research pretty much anywhere in the world.

As this unfolds Simon sees two kinds of firms that are especially endangered: small companies and those with whose primary business is serving their domestic market.

I probably have not done justice to Simon's argument in this short post, but as he laid it out in its entirely it was quite compelling. And also refreshing. Market researchers are a paranoid bunch. When we gather at conferences we spend way too much time worrying about players from other industries (management consulting, business intelligence, social networks, etc.) coming into our space and eating our lunch. The prophets of doom are always with us, whether presenting at conferences or crying out in the blogosphere. Like the Flintstones character Schleprock we seem to carry around our own little rain clouds wherever we go. What I really liked about Simon's presentation is that it gave me a break from all of that. It was about what we must do as businesses, not the do or die methodology choices we absolutely must make before lunch. It's a different perspective on the industry and while it frankly is a little scary it is at least prescriptive and actionable. A welcome relief.

MRS Annual Conference

Last week I spent two days at the MRS Annual Conference in London.  The MRS has just changed its logo which now features the words, "Evidence Matters." I took that to be a reaction to a decade of emphasis on "insight" which at times seemed to translate to "evidence optional" or at least given us a free pass on serious evaluation of the evidence at hand.  But alas, there was precious little evidence on display through most of the presentations. I'll give the presenters a break and blame the format which mostly allotted only 10 and occasionally 15 minutes to a presentation.

Nonetheless, it was an interesting two days. Most of the NewMR suspects were there--big data, social media, gamification, behavioral economics, netnography, neuromarketing.   (More on the specifics of some of those in future posts.) There also were the increasingly-common entertaining but marginally-relevant keynoters.  

If there was a big idea I think it was this:  the most important challenge for our industry is not methodology but the need to change how we work with clients. Our role has always been to explain customers to our clients. Increasingly clients want more; they want to connect with those customers.   Disintermediation probably is too strong a word but there were joint researcher-client presentations where the interaction between clients and consumers was so intimate that the audience wondered out loud if researchers might not eventually be left with little to do beyond the logistics of creating the opportunities for those interactions.  Using the focus group metaphor, it's about getting clients from behind the glass and out into the room.  And by clients I mean not just the people who commission the research but line managers from within the client company charged with implementation of the research findings. An all-day strategy session with line managers and research participants was one example of how this is being done.  All in all we saw examples of how the research process is becoming less structured, more interactive, more iterative and ultimately more innovative.  It makes understanding our clients' businesses as well as they do an absolute imperative. This is exciting stuff, even for a survey geek.

The latest online panel dust up

Gregg Peterson's post earlier this week on this blog about the Panel of Panelists at the CASRO Online Conference created quite a stir. I saw an unusual number of pageviews, there was a fair amount of retweeting of the link and other industry commentators worked a similar theme. It came on the heels of Ron Sellers's More Dirty Little Secrets of Online Panels which also created quite a stir.

What I find surprising about all of this is, well, that people are surprised. For anyone paying half attention to the quality issues of online panels the core of what Gregg reported ought to be old news. I thought that we had already worked our way through the first four stages of the Kübler-Ross model and arrived at Acceptance. Recognizing that some people are doing an awfully large number of surveys and doing it for money is the least of it. This is convenience sampling run amok.

We've seen this movie before. A former colleague once spent much of his time on the road doing focus groups with IT managers. Time after time in every city it was always the same faces. He liked to call them "the Band of Brothers." He claimed to be on a first name basis with many of them. But the widely-acknowledged professional focus group participant did not completely undermine the usefulness of that methodology.

I've long been very critical of online and quick to point out its numerous shortcomings but I also think that we are better at it today than we have ever been. To the industry's credit, we finally have come to face its flaws and are evolving a set of standards and practices to deal with them.  We have a pretty good idea of how to do a better job of managing panels and are extending that to other non-panel sample sources. We have finally admitted that our sampling strategies need to evolve beyond age, gender and regional quotas. There are some very smart people helping us think our way through this. Granted, these developments are not yet industry wide. There are still way too many practitioners making outrageous claims, too many panel companies that have not embraced the kind of transparency that's needed for buyers to make sound judgments about quality and too many clients buying by the pound. Too much DADT.

We all know the old cliché about admitting you have a problem being the first step. I thought we were there.

The end of “don’t ask, don’t tell” in online survey research?

My colleague, Gregg Peterson, attended last week's CASRO Online Research Conference and has sent me this post.

The era of "don't ask, don't tell" in the world of commercial market research may well have ended last week in Las Vegas at the annual CASRO Online Research Conference.    The purveyors of online surveyors came face-to-face with the survey taking monsters they created.  Back in our rooms, if we looked up at the big mirrored walls of our swanky, conference hotel suites, we noticed the real culprits staring straight back at us.

Hands down, the talk of last week's conference was "the panel of panelists".  These were eight people recruited at the back of a large quantitative study of web survey takers who were there to help us understand their world and to tell us what they liked and disliked about the surveys we served up to them.  They were selected from among a few hundred qualified Las Vegas respondents (the Las Vegas respondents being a sub-quota of the large national study) based initially on the quality and thoughtfulness of their open ended responses in the quantitative survey, and, at a second stage, in a follow-up interview with the panel's eventual moderator, to see how well they could articulate their respective panel taking experiences.  (Let's hope that the recruiter was unconcerned about demographic diversity, because 7 of the 8 were not employed full time, none were below the age of 30, and all were Caucasians.)

And articulate they did - all of the scary sins of our industry.  Here's the least surprising of what we learned:  Each of them were members of multiple panels – the least ambitious of them was a member of a mere four panels while the majority of them seemed to be a members of 8-10 panels. When asked who had received multiple invitations to the same survey, all hands shot up immediately. A few happily admitted to taking the same survey multiple times.  Some seemed to be aware that it might not be completely kosher to do so, while one claimed proudly to have responded to all 10 invitations.  ("It's not up to me to police you guys.")  One very articulate elderly woman suggested that panel companies should allow them to indicate when they will be on vacation, because it was "so hard to come home to 600 or 800 survey invitations."  All were very clear about the importance of incentives and one panelist admitted to taking surveys and participating in focus groups simply to pay off the mortgage on his vacation home.   And he was less than happy about the new speeder detection tools sometimes imbedded in our surveys.    "It's easy to go fast when you've seen the same thing over and over again." 

As always, there are caveats. This was, after all, a qualitative study – not even pretending to be representative like most of the surveys we report on. That being said, there was some good news here.  These folks take their survey taking responsibilities seriously.  They did surveys at least in part because they really like giving their opinions.  They seemed sincere in their pledge of honesty and good intentions about providing truthful answers to all questions including screeners.  They only wish we were better at our jobs.  The spoke of broken links, screeners that take forever, highly repetitive surveys, progress bars that clearly lie, 50 minute surveys with too small an incentive, incentives that never show up, questions without a "Don't Know" category, missing response categories in our employment questions (retired people do not want to mark themselves as unemployed) and the dread of being confronted with dense text or a "a page full of tiny little boxes". They don't think their detergent has a personality and they were certain that there aren't 60 different ways to describe a soft drink.  A few really preferred surveys that gave them a chance to provide detailed open-end responses and many articulated the pleasure of taking a visually interesting and well-constructed survey.

What was perhaps most surprising was the reaction of researchers following this session. I heard a senior sales executive of a very large panel company express shock and amazement about the volume of surveys being taken and the professionalism of the respondents on the panel.  Everyone was abuzz.  Perhaps it was just the relief of getting it out in the open once and for all.  We could suddenly talk more freely about what we feared – the life blood of our industry may be a small army of highly determined professional survey takers. 

And guess what other topic was prominently featured on this year's agenda? Routers and maximizing panel "efficiency".  In other words, let's figure out if we can get these poor souls to do a few more poorly constructed surveys. We have work to do.