ISO 20252 – Market, opinion and social research

In December we were audited for ISO compliance. After a week of staff interviews, project reviews and records inspections we were certified to the 20252 standard. Iso Good for us!

Now I recognize that there are more than few people in the industry who think ISO certification is some kind of retro thing that does not reflect how much the industry is changing. More on that in a bit. But first let me say that when it's all said and done ISO 20252 comes down to four pretty basic things:

  1. Making sure that people know how to do their jobs and can succeed in the assignments we give them.
  2. Ensuring that everyone on a project team understands their individual roles, the roles of their colleagues and how to work as a team to produce great work for their client.
  3. Being completely transparent with clients about what was done and how it was done so that they can make whatever business decisions they need to make with confidence.
  4. Minimizing the mistakes that generate rework and disappoint clients while maximizing the chances of learning from those mistakes.

Now back to those folks who say that ISO is for the dinosaurs and out of place in the fast-moving world of contemporary market research. Or that all that process ISO insists on works against the innovation that is the supposed life blood of boutiques and startups. I think that's nonsense. A well-defined infrastructure of process and procedures need not be a barrier to change. In fact, it can make it easier. In Good to Great Jim Collins observes that in great companies people work within the framework of a highly developed system that fosters what he calls "a culture of discipline." When you have a culture of discipline you spend less time managing people and more time managing the system. And when change is called for you change the system, not the people. That's also what ISO brings you. I recommend it to you one and all.

Zero defects: An admirable but elusive goal

Several years ago I was asked to write a chapter for a book called Methods for Testing and Evaluating Survey Questionnaires. So a couple of colleagues and I wrote something on testing online questionnaires. It led me to scratch the surface of the contemporary software testing literature where I learned that the industry had more or less run up the white flag on zero defects, that software has become so complicated and the competitive pressures to get releases out quickly so intense that most people had quietly given up on the idea of a first release being bug free. This struck me as somewhat analogous to what's happened in MR over the last decade: research designs have become more complex, the questionnaires to support them have followed suit, but the timelines clients insist on continue to get shorter and shorter. So questionnaires are more convoluted, there are more lines of code and more numbers to check but with less time to do it.

Of course, we all insist to our clients that we check it all and that remains the goal. But even if we have all the time we need there are two lessons I took from the software QA literature and they have to do with the priorities that should guide our approach to QA:

  1. Focus first on the most important stuff, that is, the section of the questionnaire, the lines of code and the analytic outputs that will create the biggest problem if they are wrong.
  2. Focus next on those places where there is most likely to be an error, that is, where the questionnaire or code is most complex and the numbers hardest to compute.

Then check everything else. Clients rightfully expect that every deliverable we give them be 100 percent correct. Getting there is not easy.

Staying connected to reality

I spent the better part of the last three days sitting in a room with representatives from research associations in 10 countries going through the ISO 20252 section by section if not word by word.  We had two overarching goals: (1) to make the standard independent of any specific data collection technology and (2) to incorporate the experience that comes from having certified almost 300 companies worldwide.  I confess that I also was doing my usual monitoring of Twitter and LinkedIn, imagining that I was somehow staying connected to the "real world."  That world generally doesn't have a whole lot of respect for the work we were doing in Toronto.  It thrives on the latest factoids about people's use of social media, airy statements from big CPG research buyers about creativity and insight and the futility of asking questions to learn about what people think. Maybe I'm following the wrong people, but there seems to be precious little talk about best practices, about gathering information in a disciplined way, and clearly walking clients through the steps we took to get to those insights we're delivering. You can't blame me for wondering if my time in Toronto was being well spent.

That's the point at which I reminded myself of Daniel Patrick Moynihan's famous admonition, "Everyone is entitled to their own opinion but not their own facts."  Call me naïve, but I think the primary goal of research still is to help clients make evidence-based business decisions.  Gathering that evidence in a systematic and transparent way so that others (for example,  clients) can evaluate it as we have evaluated it and follow our reasoning to the insights we're offering is essential. 

You often hear it said these days that research is part science and part art.  ISO is focused on the science part and you really have to wonder why some people find it so scary. 

ISO Certified Companies Surge in Honimichl Top 50!

The title of today's post is a prediction. That's also how Tom Anderson starts a post on his blog under the title, "CASRO ISO Certification Fails!" Here I pause, as the newsies do, to disclose that I am a member of the CIRQ Advisory Board—the nonprofit that CASRO set up to manage the ISO certification process in North America—and an occasional ANSI rep on the Technical Committee that developed and now maintains ISO 20252. But that probably makes me no more conflicted than Tom who in his post champions the FTO, a somewhat different certification program that he founded.

Tom says ISO certification is unnecessary. This strikes me as odd since the essence of the standard is to create transparency between research suppliers and their clients, the same goal Tom has for the FTO. The difference is that the FTO is concerned with just one part of the elephant, the part that goes offshore, while ISO creates transparency end-to-end, across the entire research process from proposal to final reporting and documentation. It offers the kind of transparency that lets clients know exactly how the research was done so that they can judge its quality and the weight they should give it in the business decisions they need to make. ISO makes FTO unnecessary. And, of course, FTO is self-certification while ISO relies on outside, trained auditors. "Trust but verify," to quote a now thankfully obscure American politician.

Tom also thinks ISO certification is wasteful by which I think he means lots of money poorly spent. And it's probably true that if your house is not in order and you don't have good procedures, standard ways of doing things, or effective and consistent training then it's going to cost you some money to shape up. But I expect that most of us will agree that having good processes and well-trained people working in them reduces rework and shortens cycle times, a good thing for our clients and our bottom lines. In other words, this is an investment worth making. But for those among us who have already made that investment, the cost of certification is not a major issue.

Tom deserves a lot of credit for launching FTO, but it's just one step in the right direction. There is a lot of research out there that when closely scrutinized is not what it claims to be, purports to do something that it does not do. Clients need a way to separate the wheat from the chaff and ISO will help them do that.

ISO Cometh

It looks like ISO certification finally is coming to US market research.

By way of background, in 2006 the International Organization for Standardization published a new standard, ISO 20252, for "Market, opinion, and social research." The primary impetus behind the standard was to facilitate global market research by creating a consistent standard worldwide. Equally important, in my view, is the standard's goal of "encouraging consistency and transparency in the way surveys are carried out, and confidence in their results and in their providers." The standard includes a set of definitions for the key terms used in research, requires that every step in the research process be standardized and documented, and that these materials be shared with clients on request. A key motivating principle of the standard is that standardization of processes and consistent application of them in research projects will lead to improvements in research results. Equally key is a requirement that a firm submit to a certification and ongoing auditing process to ensure that it is operating consistently within its procedures.

The industry's response worldwide has been mixed. Some countries—most notably the UK, Australia, and Mexico—have embraced it quickly and on a fairly broad scale. Other countries, like Canada, have used the standard as the basis for their own process requirements and created a sort of "ISO lite" outside of the ISO process. US companies generally have been cool to the standard, even though the Technical Committee that created the standard had US representatives who participated in its development. I confess that from time to time I was part of the process by reviewing some drafts and participating in committee meetings. Further, I admit to being an ardent ISO supporter.

Why the resistance in the US? I think there are three reasons.

First, some people are legitimately concerned about the cost. Not only do you have to pay someone to conduct the certification process, you also need to put a lot of time and effort into standardization, documentation, and ongoing monitoring. For smaller companies in particular, this could be a significant cost burden.

Second, many companies in the US were forced to do some sort of ISO 9000 certification in order to work with certain industries, most notably, the auto industry. The 9000 series is essentially a manufacturing standard that does not translate all that well to a service industry like MR. Companies forced to do 9000 often found it expensive and not especially useful.

Finally, I think there is significant misunderstanding of the 20252 standard. There are those who say that it sets the bar too low in what it requires and others who claim that it's not specific enough to be useful in promoting quality research. There is some truth here. One major problem the Technical Committee that developed the standard had to overcome was the broad variation in practice across the roughly 20 countries involved in the discussion. But the real key here is to understand that ISO 20252 is not a quality standard per se. Rather, it's a service standard that stresses the need for standardization and transparency, especially relative to clients. It may not be as precise as some people might like it terms of specifying exactly how research should be done. However, it requires that companies tell clients exactly how they are doing the work and leaves it to clients to judge whether the quality of the resulting research meets their need.

The big news in the US is that CASRO has formed a task force to study implementation. I think this is long overdue and I am excited to see it moving forward. Expect to hear more about ISO in the coming months, both here, in industry communications, and at conferences.