The Survey Geek

The limits of outsourcing

Virtually every MR company of any size that I can think of does some amount of outsourcing to offshore entities, whether to a wholly-owned sub or one of the major providers of market research services. The attraction has tended to be threefold: (1) a smart, well-educated labor force; (2) a near fanatical reverence for process; and (3) and significant cost savings. The companies that do this work provide an increasingly broad range of services including data collection, survey programming, tabulation, charting, and basic analysis. Some have become very large and there are those who believe that they may soon evolve into major, global full-service research companies in their own right.

But there are also are those who argue that outsourcing may becoming the victim of its own success, that its traditional strengths may now be weaknesses. In an interview out on cio.com Andrew Wasser from Carnegie Mellon University sees IT outsourcing as now facing two major challenges. The first is talent. As the outsourcing industry has grown the demand for top tier talent increasingly surpasses what the in-country universities can deliver. They no longer can count on getting the top decile of math, computer science and engineering graduates. MR outsources probably always had trouble competing against the Accentures of the world for top talent and likely are moving even further down the chain as the big IT firms grow. The second challenge is difficulty innovating. The strong process infrastructure that has made these companies terrific order takers and on time quality performers also makes it difficult for them solve problems that their processes were not designed to handle. And, more often than not, outsourcers see only one part of the elephant, don't have a grasp of the business problem at the core of the research and can't be the creative strategic partners that the sourcing company had hoped for.

None of this necessarily means that we have reached the limits of outsourcing. But we need to think much differently about how we structure our relationships with our outsourcing partners and be sure to be realistic about what we can get out of those relationships. We can't just keep throwing the work over the wall which unfortunately is what a lot of us have been doing. We probably need to get really serious about treating them like partners, figuring how to do a better job of integrating them more fully into our workflow, and be a good deal more realistic about what we can expect to get out of them.

MRS Annual Conference

Last week I spent two days at the MRS Annual Conference in London.  The MRS has just changed its logo which now features the words, "Evidence Matters." I took that to be a reaction to a decade of emphasis on "insight" which at times seemed to translate to "evidence optional" or at least given us a free pass on serious evaluation of the evidence at hand.  But alas, there was precious little evidence on display through most of the presentations. I'll give the presenters a break and blame the format which mostly allotted only 10 and occasionally 15 minutes to a presentation.

Nonetheless, it was an interesting two days. Most of the NewMR suspects were there--big data, social media, gamification, behavioral economics, netnography, neuromarketing.   (More on the specifics of some of those in future posts.) There also were the increasingly-common entertaining but marginally-relevant keynoters.  

If there was a big idea I think it was this:  the most important challenge for our industry is not methodology but the need to change how we work with clients. Our role has always been to explain customers to our clients. Increasingly clients want more; they want to connect with those customers.   Disintermediation probably is too strong a word but there were joint researcher-client presentations where the interaction between clients and consumers was so intimate that the audience wondered out loud if researchers might not eventually be left with little to do beyond the logistics of creating the opportunities for those interactions.  Using the focus group metaphor, it's about getting clients from behind the glass and out into the room.  And by clients I mean not just the people who commission the research but line managers from within the client company charged with implementation of the research findings. An all-day strategy session with line managers and research participants was one example of how this is being done.  All in all we saw examples of how the research process is becoming less structured, more interactive, more iterative and ultimately more innovative.  It makes understanding our clients' businesses as well as they do an absolute imperative. This is exciting stuff, even for a survey geek.

The latest online panel dust up

Gregg Peterson's post earlier this week on this blog about the Panel of Panelists at the CASRO Online Conference created quite a stir. I saw an unusual number of pageviews, there was a fair amount of retweeting of the link and other industry commentators worked a similar theme. It came on the heels of Ron Sellers's More Dirty Little Secrets of Online Panels which also created quite a stir.

What I find surprising about all of this is, well, that people are surprised. For anyone paying half attention to the quality issues of online panels the core of what Gregg reported ought to be old news. I thought that we had already worked our way through the first four stages of the Kübler-Ross model and arrived at Acceptance. Recognizing that some people are doing an awfully large number of surveys and doing it for money is the least of it. This is convenience sampling run amok.

We've seen this movie before. A former colleague once spent much of his time on the road doing focus groups with IT managers. Time after time in every city it was always the same faces. He liked to call them "the Band of Brothers." He claimed to be on a first name basis with many of them. But the widely-acknowledged professional focus group participant did not completely undermine the usefulness of that methodology.

I've long been very critical of online and quick to point out its numerous shortcomings but I also think that we are better at it today than we have ever been. To the industry's credit, we finally have come to face its flaws and are evolving a set of standards and practices to deal with them.  We have a pretty good idea of how to do a better job of managing panels and are extending that to other non-panel sample sources. We have finally admitted that our sampling strategies need to evolve beyond age, gender and regional quotas. There are some very smart people helping us think our way through this. Granted, these developments are not yet industry wide. There are still way too many practitioners making outrageous claims, too many panel companies that have not embraced the kind of transparency that's needed for buyers to make sound judgments about quality and too many clients buying by the pound. Too much DADT.

We all know the old cliché about admitting you have a problem being the first step. I thought we were there.

The end of “don’t ask, don’t tell” in online survey research?

My colleague, Gregg Peterson, attended last week's CASRO Online Research Conference and has sent me this post.

The era of "don't ask, don't tell" in the world of commercial market research may well have ended last week in Las Vegas at the annual CASRO Online Research Conference.    The purveyors of online surveyors came face-to-face with the survey taking monsters they created.  Back in our rooms, if we looked up at the big mirrored walls of our swanky, conference hotel suites, we noticed the real culprits staring straight back at us.

Hands down, the talk of last week's conference was "the panel of panelists".  These were eight people recruited at the back of a large quantitative study of web survey takers who were there to help us understand their world and to tell us what they liked and disliked about the surveys we served up to them.  They were selected from among a few hundred qualified Las Vegas respondents (the Las Vegas respondents being a sub-quota of the large national study) based initially on the quality and thoughtfulness of their open ended responses in the quantitative survey, and, at a second stage, in a follow-up interview with the panel's eventual moderator, to see how well they could articulate their respective panel taking experiences.  (Let's hope that the recruiter was unconcerned about demographic diversity, because 7 of the 8 were not employed full time, none were below the age of 30, and all were Caucasians.)

And articulate they did - all of the scary sins of our industry.  Here's the least surprising of what we learned:  Each of them were members of multiple panels – the least ambitious of them was a member of a mere four panels while the majority of them seemed to be a members of 8-10 panels. When asked who had received multiple invitations to the same survey, all hands shot up immediately. A few happily admitted to taking the same survey multiple times.  Some seemed to be aware that it might not be completely kosher to do so, while one claimed proudly to have responded to all 10 invitations.  ("It's not up to me to police you guys.")  One very articulate elderly woman suggested that panel companies should allow them to indicate when they will be on vacation, because it was "so hard to come home to 600 or 800 survey invitations."  All were very clear about the importance of incentives and one panelist admitted to taking surveys and participating in focus groups simply to pay off the mortgage on his vacation home.   And he was less than happy about the new speeder detection tools sometimes imbedded in our surveys.    "It's easy to go fast when you've seen the same thing over and over again." 

As always, there are caveats. This was, after all, a qualitative study – not even pretending to be representative like most of the surveys we report on. That being said, there was some good news here.  These folks take their survey taking responsibilities seriously.  They did surveys at least in part because they really like giving their opinions.  They seemed sincere in their pledge of honesty and good intentions about providing truthful answers to all questions including screeners.  They only wish we were better at our jobs.  The spoke of broken links, screeners that take forever, highly repetitive surveys, progress bars that clearly lie, 50 minute surveys with too small an incentive, incentives that never show up, questions without a "Don't Know" category, missing response categories in our employment questions (retired people do not want to mark themselves as unemployed) and the dread of being confronted with dense text or a "a page full of tiny little boxes". They don't think their detergent has a personality and they were certain that there aren't 60 different ways to describe a soft drink.  A few really preferred surveys that gave them a chance to provide detailed open-end responses and many articulated the pleasure of taking a visually interesting and well-constructed survey.

What was perhaps most surprising was the reaction of researchers following this session. I heard a senior sales executive of a very large panel company express shock and amazement about the volume of surveys being taken and the professionalism of the respondents on the panel.  Everyone was abuzz.  Perhaps it was just the relief of getting it out in the open once and for all.  We could suddenly talk more freely about what we feared – the life blood of our industry may be a small army of highly determined professional survey takers. 

And guess what other topic was prominently featured on this year's agenda? Routers and maximizing panel "efficiency".  In other words, let's figure out if we can get these poor souls to do a few more poorly constructed surveys. We have work to do.