Comparison Usability Evaluation Methods in a Health Information System: Heuristic Evaluation versus Cognitive Walkthrough Method

Abstract Background There are some inconsistencies regarding the selection of the most appropriate usability evaluation method. The present study aimed to compare two expert-based evaluation methods in a nursing module, as the most widely used module of a Hospital Information System (HIS). Methods The Heuristic Evaluation (HE) and Cognitive Walkthrough (CW) methods were used by five independent evaluators to evaluate the nursing module of Shafa HIS. In this regard, the number, severity and ratio of the recognized problems according to the usability attributes were compared using two evaluation methods. Results The use of the HE and CW evaluation methods resulted in the identification of 104 and 24 unique problems, respectively. The average severity of the recognized problems was 2.32 in the HE method and 2.77 in the CW evaluation method; however, there was a significant difference between the number and severity of recognized usability problems by these methods (P < 0.001). Some problems, which were associated with effectiveness, satisfaction and error, were better recognized by the HE method; however, CW evaluation method was more successful in recognizing problems of learnability, efficiency and memorability. Conclusion The HE method recognized more problems with a lower average severity. On the other hand, CW could recognize fewer problems with a higher average severity. Regarding the evaluation goal, HE method would be used to improve effectiveness, increase satisfaction and decrease the number of errors. Furthermore, CW evaluation method is recommended to be used to improve the learnability, efficiency and memorability of the system.