Getting your money's worth! Targeting resources to make cognitive interviews most effective

Cognitive interviewing has long been hailed as an effective technique to evaluate and improve survey questions. However, cognitive interviews are typically resource intensive and thus conducted on limited sets of questions and with limited sets of respondents. To be most effective, questions that are most likely to have adverse impacts on data quality should be targeted. Respondents most likely to exhibit problems with these questions should likewise be selected for testing. One way to target a subset of questions is to use available information from previous data collections to identify questions with the greatest number of quality problems (e.g. high edit or item imputation rates, greater numbers of requests for assistance answering these questions, etc.) Once a subset of questions has been identified as good candidates for cognitive testing, respondents must also be selected. Again, information from existing data sets can be used to identify characteristics of respondents most likely to exhibit problems. Data mining techniques, such as classification trees, can be used to determine the type of respondents most likely to contribute to low quality responses. These criteria can be used to select respondents for cognitive interviews. Knowing the pertinent characteristics of these respondents may also suggest useful probes that can be included in the cognitive interviews. Once questions have been revised based on the cognitive interviews, the same indicators of quality can be used to measure the improvement in data collection using the new questions. This approach has been employed in making revisions to questions on the Census of Agriculture; a case study provided will illustrate how this is an effective use of scarce testing resources.