This morning's topic is "Optimizing Health Survey Strategies." Let's be clear about one thing: rumors of the death of probability sampling are greatly exaggerated. The NHIS uses a high quality sampling frame and routinely get 95%. The NSFG gets 79%. There is a lot of money and time spent to get these kinds of outcomes and one of the key questions for this session is whether there are less expensive ways to do health research, although outside of these "gold standard" surveys.
Overall there seemed to be three basic themes: (1) use of online panels; (2) mixed mode designs; and (3) better tools for dealing with nonresponse. The online panel argument was mostly about getting access to low, make that very low, incidence populations, in this case sufferers of primary immune deficiency disease. Even very large probability samples uncover small numbers of these individuals. The study worked pretty well and it generated considerable discussion around fit-for-purpose. No one is sure how to make those decisions on a survey by survey basis but there was some agreement that the credibility of findings from nonprobability samples can be strengthened by validating against other high quality studies. Good quality probability samples with high response rates are needed for generating highly accurate estimates and for calibration, but some relaxing of traditional quality critera seems possible in other kinds of studies.
The other topic that generated a lot of discussion is imputation. One cynic suggested we just interview two people and impute the rest. As someone who occasionally moves between conferences for the two sectors of the industry—MR and government—I am always impressed with the energy and brainpower that the government side puts into dealing with the nonresponse problem. MR has sort of thrown up its hands and moved one, but the government folks are nothing if not dogged in trying to solve the nonresponse problem whether in data collection methods, imputation, or weighting. We could learn something from our government colleagues on this score.
When I first entered market research after getting my degree, my employer was worried about showing a base on N=496 in a few places in the report when we were "paid to collect N=500." I suggested imputation; he turned bright red and said, "We will never make up data!" At the time I was mortified and irritated in equal measure, but now I'm just impressed that I was working with a market researcher who knew what imputation was (even if he didn't understood what it meant). In any case, I've always though it was underused in market resarch. But then again, item missing data is a minor issue in most commercial studies as compared to coverage and response rate.
Posted by: Theo Downes-Le Guin | April 18, 2011 at 07:05 PM