When Inter-Rater Reliability Is Obtained from Only Part of a Sample.

It is erroneous to extend or generalize the inter-rater reliability coefficient estimated from only a (small) proportion of the sample to the rest of the sample data where only one rater is used for scoring, although such generalization is often made implicitly in practice. It is shown that if inter-rater reliability estimate from part of a sample is available, the score reliability for the rest of the sample data rated by only one rater can be estimated both within the classical reliability theory framework, and within the framework of generalizability theory. As intuitively expected, score reliability for the data for which only one rater is used for scoring is always lower than the score reliability for the portion of sample data for which two raters are used. A sample of published studies is provided from difference disciplines that gives inter-rater reliability coefficients obtained from a small proportion of a sample. For this sample of published studies, by applying the method discussed in this paper, the estimated score reliability is given for the data rated by only one rater. (Contains 1 table and 20 references.) (Author/SLD) ******************************************************************************** * Reproductions supplied by EDRS are the best that can be made * * from the original document. * ******************************************************************************** RATER.V1 PERMISSION TO REPRODUCE AND DISSEMINATE THIS MATERIAL HAS BEEN GRANTED BY kcio feLn TO THE EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) U.S. DEPARTMENT OF EDUCATION Office of Educational Research and Improvement EDUCATIONAL RESOURCES INFORMATION CENTER (ERIC) elf.:.:[s document has been reproduced as received from the person or organization originating it. 0 Minor changes have heen made to improve reproduction quality. Points of view or opinions stated in this document do not necessarily represent official OERI position or policy. When Inter-Rater Reliability Is Obtained from Only Part of a Sample Xitao Fan Utah State University Michael Chen University of Mississippi Running Head: Inter-Rater Reliability Please send correspondence about this paper to: Xitao Fan Department of Psychology Utah State University Logan, UT 84322-2810 Phone: (435)797-1451 Fax: (435)797-1448 E-Mail: fafan@cc.usu.edu Paper presented at the 1999 Annual Meeting of the American Educational Research Association, April 19-23, Montreal, Canada (Session # 36.14). BEST COPY AVAILABLE 2 Inter-Rater Reliability 2 Abstract It is erroneous to extend or generalize the inter-rater reliability coefficient estimated from only a (small) proportion of the sample to the rest of the sample data where only one rater is used for scoring, although such generalization is often made implicitly in practice. It is shown that if inter-rater reliability estimate from part of a sample is available, the score reliability for the rest of the sample data rated by only one rater can be estimated both within the classical reliability theory framework, and within the framework of generalizabifity theory. As intuitively expected, score reliability for the data for which only one rater is used for scoring is always lower than the score reliability for the portion of sample data for which two raters are used. We provide a sample of published studies in different disciplines that provided inter-rater reliability coefficients obtained from a small proportion of a sample. For this sample of published studies, by applying the method discussed in this paper, we provided the estimated score reliability for the data rated by only one rater.It is erroneous to extend or generalize the inter-rater reliability coefficient estimated from only a (small) proportion of the sample to the rest of the sample data where only one rater is used for scoring, although such generalization is often made implicitly in practice. It is shown that if inter-rater reliability estimate from part of a sample is available, the score reliability for the rest of the sample data rated by only one rater can be estimated both within the classical reliability theory framework, and within the framework of generalizabifity theory. As intuitively expected, score reliability for the data for which only one rater is used for scoring is always lower than the score reliability for the portion of sample data for which two raters are used. We provide a sample of published studies in different disciplines that provided inter-rater reliability coefficients obtained from a small proportion of a sample. For this sample of published studies, by applying the method discussed in this paper, we provided the estimated score reliability for the data rated by only one rater.