The Survey Geek

This is worth your time

One of the links over on the right is to an online journal called Survey Practice.  Rather than me trying to describe their editorial focus and policy it's probably best to check it out firsthand here.  I point all of this out to you because the current issue is focused on nonprobability sampling and while I've not actually read any of the pieces yet a quick peruse suggests they are worth a look.

Finally, the real issue?

This month's Research Business Report has two very interesting and related pieces. In one the editor takes the ARF ORQC to task for their somewhat languid reporting on the results of their Foundations of Quality initiative as well as the usefulness of their findings, or at least those findings that have been released to date. (In case you've not been paying attention, the ARF has spent a ton of money doing basic research on research aimed at improving our understanding of some of the long-talked about quality concerns with online panels.) At the risk of oversimplifying, I would characterize the ARF results to date as reassurance that the principal panel quality issues the industry has been most worried about over the last few years—fraudulent respondents, multipanel membership, people taking the same survey more than once, high levels of satisficing, etc.—are real but controllable, either by the panel supplier or the researcher.

Unfortunately, in the process of answering what we thought were the big questions they have raised an even bigger one. Here is one bit of the ARF data that has made it into the public domain. When comparing the incidence of smoking in survey results from 17 panels to benchmark data using traditional probability-based methods they found that (a) the three probability-based methods produced similar results while (b) the variation across panels varied by as much as 14 points. This should not surprise us. Panels are all very different in how they recruit their members and the way in which they manage those members over time. We can control the demographics, at least at the sampling stage, but the mix of attitudes and behaviors of panelists will vary substantially from panel to panel based on a wide range of things (e.g., where they were sourced, the appeal that was used, the incentive scheme, the survey topics they offer, etc.) that we still don't understand very well. In the end, the results we get for any given study are highly dependent (and mostly unpredictable) on the panel we use. This is not good news.

Which brings us to the second piece, the RBR lead story. For about the last 18 months Steve Gittleman has been doing something he calls, "The Grand Mean Project." In a truly prodigious effort, Steve has amassed survey data covering something like 90 panels worldwide and has been using those data to study variability among panels. He argues that with the move away from probability-based methods and the consistency that comes with a rigorously applied scientific approach we have lost our anchor to reality and with it any hope of using panels to get at "true values" for survey variables. Rather, the best we can hope for is that the same survey administered to the same panel multiple times will produce consistent results. Administer that survey to another panel, and you likely will get different results. Steve believes that he has developed a method that measures panel consistency over time and that you can use that method to identify the most consistent panels. In essence he is making an argument others have made to the effect that bias is not necessarily a bad thing as long as we know it's there, it's consistent over time, and we have other sources of information on the same topic that can help us correct it.

A little over ten years ago, Gordon Black, founder of Harris Interactive, asked "whether findings from huge samples of Internet respondents, coupled with sophisticated weighting processes, are as accurate as anything done on the telephone or door-to-door." More and more it seems like the answer is no.

Who really is worrying about online data quality?

The indomitable Simon Chadwick has a a little piece in the June issue of Research World in which he reports on the results of Cambiar's annual survey of the research industry--clients and suppliers. (You can get a topline of  the report from the Peanut Labs site here.)  One of the ongoing themes of this survey since Simon launched it way back in 2005 has been concern about the quality of online sample, and that concern is front and center again in 2009.  The concern has changed some over the years from one of worry about recruitment sources and procedures to those of respondent engagement, representivity, and questionnaire design.  No surprise there.  But what is something of a surprise to me at least is his finding that "Clients . . . appear to be rather less concerned about data and sample quality than research companies are."   Ummm.  It seems to me that one underlying premise of the industry frenzy on this issue has been that clients are worried about online quality and we'd better get some good answers and get them fast.  But Simon's data suggests that may  not be the case.  Further, he points out that despite concern about online quality clients use of online and even expansion into new online methods continues unabated.  But what does it mean?  Do clients not require the level of precision and certainty that we think they do when they make business decisions?  Or are the issues so arcane that they just don't understand them or care about them?

We say that MR is important as a means to help clients understand their markets and their customers.  Findings like this make we wonder how well we as researchers  understand ours.

Sometimes I am slow to catch on

I've been on vacation and literally the last thing I did before I left was to read a piece in IJMR by Martin Oxley and Brendan Light titled, "Research 2.0: engage or give up the ghost?"  As I mulled it over on the way to the airport it dawned on me that this mixed bag of issues, ideas, and problems we lump under the general heading of "online" has four distinct themes to it.

The first of these themes is related to how we adopt traditional question and answer formats to the Web.  Mostly this is about how we transition from interviewer administration to self administration and from aural to visual.  This is the space where the academic-based survey methodologists have done a lot of good work with some worthwhile contributions from MR research on research.  This is also the turf of the Flash evangelists who I cynically view as more about promoting a business model than doing good solid methodological research.

The second theme is really a major paradigm shift and is all about changing how we extract (if that's the proper word) the information we need from people.  Rather than asking respondents (again, if that's the right word) questions that require them to translate their reactions, thoughts, opinions, etc. into words it tires to provide images and symbols as the main answering devices.  That's the essence of what Oxley and Light describe.  These methods have generally been more widely promoted and used in Europe than elsewhere.

The third theme is focused on the people whom we interview (again, if that's the right word).  This is where we get into all of those arguments about representivity, professional respondents, satisifcing, etc.  There probably has been more work in this area than the others, especially in the US, but how much light is being shed remains an open question.  This has been mostly about access panels but increasingly now is about social networks. Rightly or wrongly, this has been the turf where the debate about the quality of online research has been joined.

Finally, there is the "listening" theme.  This is all about harvesting the ongoing conversation of the Web to help clients understand how their products and brands are being perceived.  Too often it's not just about listening, but also about intervening which creates an interesting dilemma for researchers around where research ends and marketing begins.

Maybe this has been obvious to other people all along, but it's been a revelation of sorts to me.  I always find it easier to understand a problem when you can compartmentalize it and look at its pieces one at a time.

No surprise: It just keeps getting worse

The latest numbers on wireless only households as measured by the NHIS have just been released.  They now estimate that as of December 2008 20.2 percent of US households have only a wireless telephone.  The 2.7 percent increase in the second half of 2008 is the largest six month increase since the NHIS has been collecting these data. 

I recently saw some estimates from SNL Financial that appear to have taken the NHIS trends and projected them forward.  Their data say that by 2013 a whopping 37 percent of US households will be wireless only. 

As I have noted in the past, the most statistically robust way of dealing with this problem is to spend the extra money and dial some cell phones.  The financial hit is not too bad as long as you only have to do limited dialing.  But as the proportion of cell only Rs increases, so do the costs.  While there is always the option of mixing in wireless only Rs from an online panel that is a journey into a statistical no man's land.