The Survey Geek

Online samples: Paying attention to the important stuff

Those of you who routinely prowl the MRX blogosphere may have noticed a recent uptick in worries about speeders, fraudulent respondents, and other undesirables in online surveys. None of this is new. These concerns first surfaced over a decade ago, and I admit to being among those working the worry beads. An awful lot has changed over the last 10 years, but it seems that not everyone has been paying attention.

Yesterday, my buddy Melanie Courtright at Research Now reached her I’m-not-going-to-take-it-any-more moment and posted an overview of what are now widely accepted practices for building and maintaining online sample quality. Most this is not new, nor is it unique to Research Now. If you are really worried about this stuff, choose your online sample supplier carefully and sleep at night. ESOMAR has for many years provided advice on how to do this (select a supplier, not sleep at night).

Of course, none of this guarantees that you are not going to have some speeders sneak into your survey who will skip questions, answer randomly, choose non-substantive answers (DK or NA), etc. Your questionnaire could be encouraging that behavior, but let’s assume you have a great, respondent friendly questionnaire. Then the question is, “Does speeding with its attendant data problems matter?”  The answer is pretty much, “No.” It may offend our sensibilities but the likely impact on findings is negligible. Partly that’s because we seldom get a large enough proportion of these “bad respondents” to significantly impact our results, but also because their response patterns generally are random rather than biased. See Robert Greszki’s recent article in Public Opinion Quarterly for a good discussion and example.

The second iteration of the ARF's Foundations of Quality initiative also looked at this issue in considerable detail and offered these three conclusions:

• For all the energy expended on identifying those with low quality responses, they may make less of a difference in results than focusing more clearly on what makes for a good sample provider.

• Further, when sub-optimal behaviors occur at higher rates, they generally indicate a poorly designed survey – some combination of too long, too boring, or too difficult for the intended respondents. Most respondents do not enter a survey with the intention of not paying attention or answering questions in sub-optimal ways, but start to act that way as a result of the situation they find themselves in.

• Deselecting more respondents who exhibit sub-optimal behaviors may increase bias in our samples by reducing diversity, making the sample less like the intended population.

The irony in all of this is that the potential harm caused by a few poor performing respondents pales in comparison to the risk of using samples of people who have volunteered to do surveys online, especially in countries with low Internet penetration. There is the widely-accepted belief in the magical properties of demographic quotas to create representative samples of almost any target population. No doubt that works sometimes, but we also know that, depending on the survey topic, other characteristics are needed to select a proper sample. What characteristics and when to use them remain open questions. Few online sample suppliers have proven solutions and outside of academia little effort is being put to developing one.

AAPOR gets it wrong

Unless you’ve been on vacation the last couple of weeks chances are that you have heard that The New York Times CBS News have begun using the YouGov online panel in the models they use to forecast US election results, part of a change in their longstanding policy of using only data from probability-based samples in their news stories. On Friday, the American Association for Public Opinion Research (AAPOR) issued a statement essentially condemning the Times (and its polling partner, CBS News) for “rushing to embrace new approaches without an adequate understanding of the limits of these nascent methodologies” and for a lack of “a strong framework of transparency, full disclosure, and explicit standards.”

I have been a member of AAPOR for almost 30 years, served on its Executive Council, chaired or co-chaired two recent task force reports, and, in the interest of transparency, note that I unsuccessfully ran for president of the organization in 2011. AAPOR and the values it embraces have been and continue be at the center of my own beliefs about what constitutes good survey research. That’s why I find this latest action to be so disappointing.

The use of non-probability online panels in electoral polling is hardly “new” or “nascent.” We have well over a decade of experience showing that with appropriate adjustments these polls are just as reliable as those relying on probability sampling, which also require  adjustment. Or, to quote Humphrey Taylor,

The issue we address with both our online and our telephone polls is not whether the raw data are a reliable cross-section (we know they are not) but whether we understand the biases well enough to be able to correct them and make the weighted data representative. Both telephone polls and online polls should be judged by the reliability of their weighted data.

There is a substantial literature stretching back to the 2000 elections showing that with the proper adjustments polls using online panels can be every bit as accurate as those using standard RDD samples. But we need look no further than the 2012 presidential election and the data compiled by Nate Silver:

 

I don’t deny there is an alarming amount of online research that is just plain bad (sampling being only part of the problem) and should never be published or taken seriously. But, as the AAPOR Task Force on Non-Probability Sampling (which I co-chaired) points out, there are a variety of sampling methods being used, some are much better than others, and those that rely on complex sample matching algorithms (such as that used by YouGov) are especially promising. The details of YouGov’s methodology have been widely shared, including at AAPOR conferences and in peer-reviewed journals. This is not a black box.

On the issue of transparency, AAPOR’s critique of the Times is both justified and ironic. The Times surely must have realized just how big a stir their decision would create. Yet they have done an exceptionally poor job of describing it and disclosing the details of the methodologies they are now willing to accept and the specific information they will routinely publish about them. Shame on them.

But there also is an irony in AAPOR taking them to task on this. Despite the Association’s longstanding adherence to transparency as a core value they have yet to articulate a full set of standards for reporting on results from online research, a methodology is that is now almost two decades old and increasingly the first choice of researchers worldwide. Their statement implies that such standards are forthcoming, but it’s hard to see how one can take the Times to task for not adhering to them.

My own belief is that this is the first shoe to drop. Others are sure to follow. And, I expect, deep down most AAPORites know it. AAPOR is powerless to stop it and I wish they would cease trying.

I have long felt that AAPOR, which includes among its members many of the finest survey methodologists in the world, would take a leadership role here and do what it can do better than any other association on the planet: focus on adding much needed rigor to online research. But, to use a political metaphor, AAPOR has positioned itself on the wrong side of history. Rather than deny the future, I wish they would focus on helping their members, along with the rest of us, transition to it.

 

 

A bad survey or no survey at all?

For a whole lot of reasons that I won’t go into online privacy suddenly is front and center, not just in the research industry, but in the popular press as well. The central message is that people are “concerned,” but about what exactly and by how much, well the answers there are all over the map. One of the few clear things about this whole debate, if that’s what it is, is the ongoing misuse of online surveys to describe what is going on.

I am hard pressed to think of anything sillier than using online surveys to help us understand attitudes about online privacy. Think about it. You have a sample of people who have signed up to share their personal behavior, attitudes, and beliefs in online surveys. What in God’s name could possibly make us think that these online extroverts, this sliver of the population, could possibly represent the range of attitudes about online privacy among “consumers” as generally alleged?  If ever there was an example of an issue where online is not fit to purpose, this is it. Yet these surveys are churned out weekly, generally to serve the commercial interests of whoever commissioned them, and often widely cited as some version of the truth.

To quote H. L. Mencken, “A newspaper is a device for making the ignorant more ignorant and the crazy crazier.” Sometimes it feels like online surveys serve a similar purpose.

 

Pleeezz!

Today’s update from Research-live.com has this headline: Online trackers not optimised for mobile could 'compromise data quality.' It goes on to explain:

GMI, which manages more than 1,000 tracking studies, claims that online trackers that haven’t been optimised for mobile platforms may exclude this growing audience, which could lead to a drop in data quality, reduced feasibility and the possibility of missing whole sections of the required population from research.

Let me be clear. I don’t disagree that online surveys need to be optimized for mobile and that the numbers of unintentional mobile respondents (aka UMRs) is large and growing. But a warning from an online panel company that scaring away UMRs may be leading to a drop in data quality because of “the possibility of missing whole sections of the required population from research” just drips with irony.

Let’s start with the fact that online research, at least in the US, by definition is excluding the roughly 20% of the population that is not online. Research using an online panel of, say, two million active members is excluding about 99% of the adult population. As the industry has moved more and more to dynamic sourcing it’s hard now to know how big the pool of prospective online respondents is, but it’s a safe bet that that the vast majority of US adults are missing, and not at random.

Surely, if we have figured out a way to deal with the massive coverage error inherent in the online panel model, we can handle the mobile problem.

I suspect that the real issue here is feasibility, not data quality. Just as the now near-universal use of routers is about inventory management rather than improved representativeness. I wish that online panel companies would spend more time trying to deal with real data quality issues like poor coverage and inadequate sampling methods, but that’s only going to happen if their customers start demanding it.

Thinking fast and slow in web survey design

I am a huge fan of Jakob Nielsen's work on web usability.  He has a post out this week--"Four Dangerous Navigation Approaches that Can Increase Cognitive Strain"--that puts web usability into a system 1/system 2 framework.  As I've said many times before, I believe that his research on web usaiblity has important implications for web survey design. 

In his post Nielsen offers evidence for a principle I have long aruged is important in web survey design: unfamiliar answering devices and complex instructions absorb cognitive energy and distract from the key task of simply providing an answer to a question. I'm not going to rehash Nielsen's full post here, but encurage you to follow the link and have a read for yourself.  You may want to pay special attention to dangerous navigation approach number four: "Fun" tools that become obstacles.