Scott Keeter of the Pew Research Center has an interesting research note in the current issue of Public Opinion Quarterly (Spring, 2006). He looked at whether the exclusion of cell phone only voters from telephone polls for the 2004 elections created any bias in the estimates. The answer: No. He points out that cell phone only households are primarily a life style choice of the 18-24 and 25-29 age groups. But , at least where political attitudes are concerned, they are not all that different from others in their age cohorts. So while the undercoverage is a potential problem, weighting by age seems to correct for it. Good news, for now.
Posts from March 2006
Last week I spent a few days in Washington at an annual conference that brings together government survey people and the contractors who do much of their work. The conference is jointly sponsored by the Bureau of the Census and the Bureau of Labor Statistics. It's mostly a lot of show and tell, a good way to get a sense of what's going on at a nuts and bolts level in that sector of the industry. A few observations:
- The term "big iron" used to be used to refer to very large mainframe systems with lots of raw processing power. In many ways the folks in this sector continue to have a "big iron" mentality despite the disappearance of the mainframe. Their data collection and management systems are big, complex, and expensive. They are constantly on the track of completely integrated solutions. Fast moving and nimble they are not, but comprehensive and able to deal with endless detail and individual variations in survey data is what they do best. Their approach reflects the requirement of very high response rates and the need to capture very detailed and comprehensive information.
- Good data collection software continues to be the bane of this sector. The perceived need for complexity that drives their system development has also led them to eschew most of the commercial systems. And so they rely on vendors who develop specifically for this market, a market that has relatively few but still very large customers. In this setting major shifts in computing paradigms wreak havoc. And so, a DOS-based system called CASES died out because the vendor lacked the money and talent to transition to Windows. The current standard is a system called Blaise from a government agency (Statistics Netherlands) but it continues to struggle even now at producing a serviceable Web component. Contractors who adopt other systems have a difficult sell to convince clients that their systems can do the job.
- There are still data collection systems being used that either are general systems developed by the agency or survey-specific systems developed to meet the requirements of just one study. The latter is not as irrational as it may seem because questionnaires in that sector tend to have very long life cycles. In commercial MR, questionnaires change all the time, project by project.
Overall, this was a good reminder of just how different that business is. As we do more of it we are going to need to recognize those differences and adjust our approach accordingly.