What is gained and lost when using methods other than empirical testing

Traditional laboratory usability testing is frequently not performed due to a company's lack of funds, planning, or human factors expertise. Consequently, there is increasing interest in finding alternative usability testing methods that are easier and cheaper to implement than traditional laboratory usability testing. Recent studies are beginning to study and compare such techniques. These methods include Heuristic Evaluation (Nielsen and Molich, 1990), and Cognitive Walkthrough (Polson, Lewis, Rieman, & Wharton, 1990). For Heuristic Evaluation, Nielsen (1991) found that human-factors Experts were the best at finding an interface's usability problems, especially Experts who were also expert in the interface domain. Desurvire, Lawrence, & Atwood (1991) found Experts' evaluations were the most reliable, and their best guess predictions were predictive of laboratory performance. Karat, Campbell, & Fiegel (1992) similarly found that heuristic results were reliable and significantly predictive of laboratory data, yet empirical laboratory testing identified four and five times as many problems. Jeffries, Miller, Wharton, and Uyeda (1991) found that via Heuristic Evaluation, more severe problems were found than with laboratory testing or the Cognitive Walkthrough. This comparison study did, however, only utilize Experts in the heuristic condition, and System Designers in the Cognitive Walkthrough.