Novices Perform Like Experts on a Closed Card Sort but Not an Open Card Sort

Developing good psychological measures benefits from the input of content experts. For many constructs or domains, however, who constitutes an ‘expert’ might be ill-defined. Novices—such as students, customers, or co-workers—may possess the same knowledge as experts. Moreover, as convenience samples, novices are more readily available and less costly. Card sorting is a technique frequently used in human factors to elicit expert knowledge. This study compared novice and expert performance on a card sort task under two conditions, an open sort and a closed sort. Because the closed sort offered a category structure for sorting, it was predicted and found that novices in the closed sort tended to match expert sorting results more closely than in the open sort. The structure also made novice solutions better approximate the expert open sort. This suggests that novices can be useful in the follow-up stage of measure development, but not in the initial stage.