The development and validation of a methodology for conducting formative evaluation of educational expert systems

Educational expert systems must undergo rigorous evaluation throughout the development process in order to ensure a high quality product. Evaluation can take two forms: formative and summative. Formative evaluation begins early and focuses directly on improving the quality of the product. Summative evaluation occurs near the end of development and seeks to demonstrate that the product performs as designed. The development of expert systems to solve problems within education is a recent solution. As a result, methodology to guide the evaluation process, especially formative evaluation, is limited. The developers of Classroom Advisor, an expert system to assist teachers of students with severe handicaps make data-based decisions about functional communication programs, recognized that systematic formative evaluation needed to take place. The codification of an extensive knowledge base in the area of functional communication required documentation that the knowledge base was complete and accurate. Thus, the purpose of this research project was to developed, field test, evaluate, and determine the feasibility of a methodology for formative evaluation to determine content (i.e., face) validity. Four modules from the knowledge base in Classroom Advisor were translated into a questionnaire with a Likert rating scale. The questionnaire was first sent to a panel of six human experts for review. Based on statistical analyses of the ratings and the experts' written comments, the modules were revised. The revised modules were translated into a second questionnaire, which was reviewed by the original panel of six experts and a second panel of three human experts. The results of the field test of the formative evaluation methodology supported its use to verify the content of a knowledge base and guide the revision process. In all four modules, content validity increased following revision. Additionally, data analyses confirmed a high level of reliability in the experts' ratings and a high level of agreement among the experts and between the two panels of expert judges. Social validation data also suggested that the formative evaluation methodology was a viable technique for improving an educational expert system during development.