The Survey Geek

Well said

I'm catching up with some reading on a flight to DC and have come across an interesting interview in the May issue of Research World. While interviewing Obama's pollster, Joel Benenson, Simon Chadwick asks, "Is it all about how you analyze the data?" Benenson's response is especially eloquent:

"Look, if you write a good survey, you have written a survey to produce data that's going to be impactful in terms of your analysis and in terms of the decisions that you're trying to make. You have some hypotheses when you draft the survey, and you develop the best way to test those hypotheses. I view the questionnaire as an art form. That's where you really lay down the architecture of what your data could look like, and if you've done it well you're going to have a very rich data set."

Just yesterday I turned down an opportunity to be interviewed by someone about DIY surveys. As I said at the time, "I don't have any data to show how bad they are; mostly I just emote about it." My argument against them generally comes down to this: well-trained researchers add value because they know how to write good questions and good questionnaires. Benenson has said it so much better.

Unfortunately, it's not just about DIY. The MR industry has become so fixated on the analytics on the back end that we seem to have lost sight of the importance of how we get the data to begin with. Simon's question reflects that bias; Benenson's answer sets us straight.

Sometimes I am slow to catch on

I've been on vacation and literally the last thing I did before I left was to read a piece in IJMR by Martin Oxley and Brendan Light titled, "Research 2.0: engage or give up the ghost?"  As I mulled it over on the way to the airport it dawned on me that this mixed bag of issues, ideas, and problems we lump under the general heading of "online" has four distinct themes to it.

The first of these themes is related to how we adopt traditional question and answer formats to the Web.  Mostly this is about how we transition from interviewer administration to self administration and from aural to visual.  This is the space where the academic-based survey methodologists have done a lot of good work with some worthwhile contributions from MR research on research.  This is also the turf of the Flash evangelists who I cynically view as more about promoting a business model than doing good solid methodological research.

The second theme is really a major paradigm shift and is all about changing how we extract (if that's the proper word) the information we need from people.  Rather than asking respondents (again, if that's the right word) questions that require them to translate their reactions, thoughts, opinions, etc. into words it tires to provide images and symbols as the main answering devices.  That's the essence of what Oxley and Light describe.  These methods have generally been more widely promoted and used in Europe than elsewhere.

The third theme is focused on the people whom we interview (again, if that's the right word).  This is where we get into all of those arguments about representivity, professional respondents, satisifcing, etc.  There probably has been more work in this area than the others, especially in the US, but how much light is being shed remains an open question.  This has been mostly about access panels but increasingly now is about social networks. Rightly or wrongly, this has been the turf where the debate about the quality of online research has been joined.

Finally, there is the "listening" theme.  This is all about harvesting the ongoing conversation of the Web to help clients understand how their products and brands are being perceived.  Too often it's not just about listening, but also about intervening which creates an interesting dilemma for researchers around where research ends and marketing begins.

Maybe this has been obvious to other people all along, but it's been a revelation of sorts to me.  I always find it easier to understand a problem when you can compartmentalize it and look at its pieces one at a time.

Public Choices in Changing Times

For the last couple of days I have been at the AAPOR Annual Conference down in hot, humid, and generally unpleasant Hollywood, FL.  For as long as I have been a member of AAPOR there has been a tradition of having a plenary session the night before the conference actually begins and on Thursday night we had two excellent speakers: Paul Donato from The Nielsen Company and Ken Prewitt from Columbia University. Paul is a veteran market researcher who has held senior research posts in a number of MR companies and Ken has led a variety of government and non-for-profit organizations including the US Bureau of the Census, NORC, and the Ford Foundation.  These guys come from different worlds and their presentations showed it.

Paul's talk was all about the transition from traditional survey research to a digital world of electronic measurement.  He took us along a progression of methods from asking (surveys, including online), listening (social networks, co-creation, crowd sourcing, harvesting blogs, etc.) and watching (principally ethnography).  I had heard similar talks before, but for many AAPORites this was new and challenging stuff.

Ken Prewitt almost immediately raised the issue of representation.  As a former Director of the Census he is attuned the role that survey data play in the formulation of public policy in a democracy.  He underscored the importance of reaching everyone, something the digital methods that Paul was describing do not do.  The movement that he foresees away for surveys was mostly about use of government agency and program administrative records.  This has been talked about for most of my career in survey research, but we have made precious little progress.

I found myself agreeing with both of them.  In the discussion they more or less did the same.  The term "fit-for-purpose" never came up but that ultimately was the issue.  It was good dialog of the sort we don't often see.

More comparisons of Web to other methods

I am finally getting around to wading through the mother lode of academic research noted in an earlier post way back at the beginning of March.  The special POQ issue has two articles, one looking at Web versus face-to-face and the other comparing CATI, Web and IVR.  The results are not particularly surprising, but it's nice to see one's suspicions confirmed with well-designed and executed research.

Dirk Heerwegh and Geert Loosveldt report on results from a survey in Belgium designed to assess attitudes toward immigrants and asylum seekers. They put considerable effort into designing both the Web and face-to-face survey based in Dillman's unimode construction principles.  In other words, they worked hard at making the two surveys as comparable as possible rather than optimizing each to its own mode.  Their results are pretty convincing.  The Web survey produced a higher rate of "don't know" responses, more missing data, and less differentiation in scales.

Frauke Kreuter, Stanley Presser, and Roger Tourangeau looked at social desirability bias across three methods--one with an interviewer (CATI), one without an interviewer (Web), and one with sort of an interviewer (IVR).  They drew a sample of University of Maryland alumni and asked a variety of questions about academic performance and post graduation giving.  They were able to verify the respondent's answers against university records.  In essence, they were able to tell who was telling the truth and who was not.  As with Heerwegh and Loosveldt, the results are pretty much what we would expect.  Web reporting was the most accurate and CATI the least with IVR generally somewhere in the middle.

So there you have it.  We used to like to say that "the Internet changes everything."  Well, it does not appear to have changed some basic principles of survey research.

Yes, but . . .

Catching up on some things that have been sitting around my inbox I came across this little tidbit from the Q&A part of a Peanut Labs Webinar in which the company pitched its social network sample product. Someone asked, "How do you ensure "Representative" sample from these social media outlets. I was more than a little surprised at their answer:

"[The sample] Peanut Labs delivers is "representative" of Internet users that belong to social networks. We cannot make claims beyond that – just as no other online sample provider can claim that their samples are representative of anyone other than people who want to join panels."

This sort of truth in advertising has been in short supply in our industry for some while, and not just in all the buzz about social networking and social media. To no small extent the disappointment that many clients have come to feel about online has its roots in the extravagant claims often made by panel companies in particular, but also online evangelists in research agencies whose business models were built around online methods.

But these moments of truth speaking can be fleeting and the PL spokesperson went on to say:

"However, research-on-research studies to be released early on in the new year show that Peanut Labs sample differs very little from offline and online sample in terms of behavior or attitude. The one exception is in their attitude to online advertising, with which they are more intensely involved."

OK, so nobody is perfect.