When Classical Measurement Theory Is Insufficient and Generalizability Theory Is Essential.

Most training programs in education and psychology focus on classical test theory techniques for assessing score dependability. This paper discusses generalizability theory and explores its concepts using a small heuristic data set. Generalizability theory subsumes and extends classical test score theory. It is able to estimate the magnitude of multiple sources of error simultaneously, unlike classical theory, which enables only a single source of error to be considered at one time. Generalizability theory forces us to see that it is scores, not the tests themselves, that are reliable. Generalizability studies are the initial round of analyses that generate variance components for the sources of error in the study. Design studies use these variance components to answer questions about alternative measurement protocols. Generalizability analyses distinguish between decisions made in the context of cutoff scores (absolute decisions) and those that consider relative standing. An example involving 10 people who have taken a 4-item test each of 3 times illustrates application of generalizability theory. Six tables and two figures present details of the analysis. (Contains 11 references.) (SLD) *********************************1.--A********************************* * Reproductions supplied by EDRS are the best that can be made * * from the original document. * ***********************************************************************