Computing composite scale scores for accountability: A validation study of Nebraska's district evaluation model

Many states use a statewide assessment strategy to evaluate districts on a common measure. Because districts in Nebraska are not measured on a common instrument, comparisons are much more difficult. This study examined an evaluation strategy for a state accountability model that considered school districts uniquely and classified them into an overall rating system (SPR) based on a combination (CSS) of salient factors. Classification decision consistency was analyzed to determine the appropriate model. Student performance and non-cognitive indicator data for three grade levels and two content areas from the 67 school districts in the state of Florida were used. Data were also obtained from a Nebraska educational advisory committee. Regression and multiattribute utility theory (MAUT) methods were used to generate appropriate weights for model contributions. Analyses comparing classification decision consistency for three mathematical models were conducted using Spearman rank order correlations for composite scale scores (CSS) and kappa statistics for school performance ratings (SPR) classifications. Results show that there is a high level of agreement between the three mathematical models considered for the study indicating that the model that is easiest to understand and communicate should be recommended. The results lead to a discussion of the implications of the models for the Nebraska accountability system.