The Survey Geek

Next/Previous or Previous/Next

As regular readers (if there are any) know, I am a big fan of Jakob Nielsen, the Web usability guru.  In this week's alert he takes up the fascinating topic of the order of buttons in a dialog box, specifically, should we display OK and then Cancel or Cancel and then OK.  His conclusion: "In cases like this, it often doesn't matter what you do. Either choice has good arguments in its favor, and no choice is likely to cause usability catastrophes. It might save some users 0.1 seconds if you pick the 'right' choice for certain circumstances, but it's simply not worth it to conduct sufficiently elaborate research to find out what that choice is."

I bring this up because it reminds me of the mini-controversy around the order of navigation buttons in a Web survey.  Should it be Next and then Previous or Previous and then Next?  In our case we did the research and summarized it in a white paper posted on the Mix way back in April of 2006 and soon on the company Web site.  We reached pretty much the same conclusion as Nielsen: it doesn't matter. All that matters is that you are consistent.  Amen.

"It's a Bright Sunshiny Day"

This report from AAPOR filed by Colleen Carlin (with apologies to Jimmy Cliff).

Finally, a sunshine-filled day here at AAPOR and maybe some good news for survey researchers…

Doug Rivers of Stanford University presented a paper on professional respondents in web surveys; having received the data only 24 hours prior to his presentation, in-depth analyses of the data are pending.  A professional respondent is defined as someone with frequent survey participation, multiple panel membership, paid to take surveys, provide inaccurate answers and exhibit fraudulent behavior.  A long, uninteresting survey was administered to respondents from four online panels.  The study was limited to the state of Delaware in an attempt to maximize panel overlap.  Traps were imbedded in the questionnaire as a way to capture inconsistent responses and 'cookies' were used to track the same respondent across panels.  Numbers indicate that panel overlap can be as high as 25 percent.  Panel membership for professional and non-professional respondents was as follows:

Number of Panels	Non-professional	Professional
1	70.7%	14.4%
2-5	23.4%	40.8%
6-10	5.2%	41.6%
More than 10	0.7%	3.2%

Professional respondents tended to be female, lower education, older and come from low income households.  As would be expected, professionals complete surveys faster and are less likely to report positive responses to industry screeners.  The good news is that despite the presence of professional respondents a comparison with non-professional respondents revealed no difference in straightlining behavior, falling into 'traps', no difference in 'select all' screeners and no difference in a discrete choice experiment embedded within the survey.  Doug had a lot of data to cover in his presentation and a lot of the findings went by too quickly to record.  I have requested a copy of his presentation and will be writing up a more in-depth summary for use at Market Strategies. 

The rest of the news from the day was not as good, but I'm going to leave things on a positive note for now, but look for a pending post on why list-assisted RDD really only covers 60 percent of the population.

Pain Perdu is Not for Everyone

A report from AAPOR filed by Dan Zahs.

No weather reports here as I was on cultural duty. In New Orleans, due to their French heritage, French toast is known as pain perdu--"those French, they have a different word for everything "(S. Martin, 1977).

The cell phone only (CPO) and the cell phone mostly estimates, although large in magnitude, only tell us part of the story. We also need to know how different these members are in order for us to evaluate potential bias. Several papers had information that addressed this issue. A paper from some folks at the Pew Research Center found some differences between the cell phone only population compared to the standard RDD population. However, these differences disappeared with the application of demographic weights. Another Pew paper and study found more significant differences when looking at  technology related issues. As will come as no surprise the cell phone only population    had higher use of various technological products or services and had more positive attitudes about technology in general. These differences remain even after adjusting for demographics. A paper from Macro International found differences in regards to health items.  Furthermore, some of these differences were small looking, small changes can have large implications in the public health area. With these three papers we see what is a common theme-- there are sometimes differences between the CPO population and the RDD population depending on the topic at hand.

More Ado About Cell Phones

A report form AAPOR filed by Collen Carlin.

Another cloudy, rainy day here at AAPOR in New Orleans and another day of cell phone discussions to attend.    Yesterday we heard further evidence that the cell phone only population is growing and today there was information on the changes to this population and ways to include cell-only people in a study in a more efficient manner.

The folks at U of M and Gallup used the Gallup panel (itself RDD – landline recruited) to conduct a mail/web study to examine the characteristics of people who had recently switched to cell-only (within the last year) to those people who made this switch prior to that time period.  The results indicate that those who made the switch to cell only recently tend to be older, married, and living in the Midwest when compared to those who have been cell only longer.  Conclusions are that the demographics of the cell only population maybe shifting away from the young, urban, lower income people that we previously knew to comprise this group. 

Next up was a study done by the folks at CBS that was akin to the work Mitofsky and Waksberg did to develop the method for RDD.  If we accept that the cell only group is creeping into a greater proportion of the population we need to investigate ways to include this group that are more economical.  The authors took a list of known cell phone, known non-cell phone and undetermined cell phone numbers and added +45 to the digit within the 100 number blocks.  The purpose was to determine if there are 100 blocks of numbers with higher proportions of working cell phone numbers, much as we find with landline numbers.  This allows us to more efficiently target working telephone numbers within an RDD method.  For the numbers that were modified from the known cell phone and undetermined numbers, approximately 75 percent were determined to be working cell phone numbers, 20 percent were non-working and 4 percent were unknown.  The numbers from the known non-working cell phone group that were modified yielded 42 percent working cell phone, 58 percent non-working cell and 0 percent unknown numbers.  The numbers in the same 100 block as the known working cell phone numbers were significantly more likely to also be working cell phone numbers, but non-working cell phone numbers did not necessarily denote an entirely bad number block.  If we need to start including cell phone numbers in our telephone surveys than it is useful to know ways to more efficiently target these numbers.

Results from another study presented explored the relationship between those people who intend to switch to cell only within the next year and those people who have gone cell only already.  This small study found similarities within these groups which could provide some information for weighting purposes. 

Waiting for the Number

A report from AAPOR filed by Colleen Carlin:

At 10 am this morning in the cloudy city of New Orleans a packed crowd of survey enthusiasts gathered in a small room, all waiting to hear 'the number'.  'The number' refers to the estimate put out twice a year by Stephen Blumburg and associates from the National Interview Health Survey.    The NIHS is a large scale, face-to-face survey that includes questions about telephone usage.  It has been invaluable in helping to estimate the percentage of cell only households in the United States.

The estimate from the last six months of 2007 is that 14.5 percent of adults (more than 32 million people) are wireless only; this is up from 12.6 percent for the first 6 months of 2007.  If you prefer to think of things in terms of households the number is 15.8 percent.  This means that nearly one out of every six homes in American is only reachable through a wireless telephone.  As survey practitioners who rely heavily on telephone methodologies this number poses an increasing problem.  But wait… it gets worse.  There is now a group of people termed 'wireless mostly' that we need to worry about.  These are the people who have a both a landline and cellular telephone, but who receive all or almost all calls on their cellular phones.  I have a good friend from school that recently got married, had a child and bought a house.  As a survey practitioner I was thrilled to hear that she had converted from one of those nasty 'wireless only' households into a dual cell and landline household upon purchasing her first home.  My elation ended when I tried to call her on her landline one day.  After several unsuccessful attempts to reach her I finally tried her cell phone.  She told me that she stopped answering her landline after the first few weeks due to the frequent sales calls she receives at that number.  Instead she uses her landline only for out-going calls when she is at home.  According to the numbers from the NIHS 22.3 percent of households with both cellular phone and landlines are considered 'wireless mostly'.  These people are nearly as difficult (although not impossible) to reach in a RDD survey as wireless only people.  Blumburg estimates anywhere from 16.4-30.4 percent of the adult population can not be contacted by a non-cell phone number.

One suggestion proposed to deal with this issue is through post survey adjustments.  Can we try to account for wireless only people by using the estimates from those wireless mostly people who can be reached by landline? Results from wireless only respondents were compared to wireless mostly and everyone else.  For most of the demographic and health variables the wireless mostly respondents were more similar to the wireless only people then the rest of the population.  In other words, while the numbers didn't match up exactly they were moving in the right direction.  It is possible that wireless mostly respondents who can be reached by a landline could represent the wireless only people in a phone survey.  It is also possible that the wireless mostly adults who respond to a landline survey might be more similar to others with landlines than to the wireless only population, meaning they would not be a good proxy.  As with all things survey related, more research is needed.