Using cognitive interviewing techniques in workplace settings

Surveys are a commonly used measurement method in organisationalresearch. They're popularity can be attributed to their relative economy,especially when self-administered, and the ease with which they can becontrolled from remote locations using mail, email or telephone. Further,surveys are the only method of observation that can describe thecharacteristics of a large population. This increases the likelihood ofachieving a large sample, thus increasing the power of any statisticalanalysis undertaken using the subsequent data, increasing the likelihoodof achieving statistically significant results even when analysing multiplevariables, and improving the generalizability of those results. However, surveys are not without their weaknesses. Survey respondentsmay experience problems with comprehension as they fail to understandwhat the question Is asking, interpret the questions in divergent ways,experience difficulty recalling information with the desired degree ofaccuracy, or offer the wrong type of response as the questions fail toelicit the type of information the survey was designed to draw out.Respondents may also feel social pressure to respond in a way which isnot the most honest or accurate, or they may consider the surveyformat cumbersome, leading them to deliberately skip items. Theseissues have the potential to diminish the survey Instrument's reliability,its ability to yield the same findings on repeated applications, and itsvalidity, or the extent to which it accurately reflects or assesses theconcept the researcher is attempting to measure. For these reasons the initial planning, development' and testing stage ofthe survey design process Is crucial. One technique that has becomeIncreasingly prominent when designing and developing surveys iscognitive interviewing (Beatty & Willis, 2007). Cognitive interviewingwas developed by survey methodologists and psychologists to evaluateand prevent sources of error in survey questionnaires. It does so byfocusing on respondent's cognitive processes as they answer surveyquestions; the covert, usually hidden processes as well as the overt,more readily observable processes. Cognitive interviewing helps identifyresponse errors the respondent may commit by misinterpreting thequestion, failing to recall crucial information, or reporting with socialdesirability response bias (Desimone & Kerstin Carlson Le, 2004).Further, cognitive interviewing allows the evaluation of the overallquestionnaire for structural or logical problems. This information canthen be used to modify the survey Instrument, reduce measurementerror, and improve its validity. In this chapter we will briefly discuss the theory behind the use ofcognitive interviewing in survey development and the availabletechniques before demonstrating, with real data, how this approachimproved an existing survey instrument designed to assess the costs ofill-health in the working population. Specifically, cognitive interviewingwas used to pilot a survey instrument within a small sample ofmanagers. By providing in-depth information and background theoryabout the survey being developed, and the importance of the concepts itis attempting to measure, we aim to highlight the value of usingcognitive interviewing techniques during survey development. We alsoaim to demortstrate the value of such techniques within a managementsample.