Item Parameter Estimation Errors and Their Influence on Test Information Functions

For test developers working within an item response theory framework, the concepts of item and test information are important and useful. Unfortunately, errors in item parameter estimates have a negative impact on the accuracy of item and test information functions. The estimation errors may be random; but because items with the higher levels of discriminating power are more likely to be selected for a test, and these items are most apt to contain positive errors in their item parameter estimates, item information functions and corresponding test information functions tend to be inflated in relation to their true values. The purpose of this article was to investigate the impact of this "capitalization on chance" in item selection on the accuracy of test information functions associated with the three-parameter logistic model. Two variables seemed especially important in determining the size of the impact: (a) examinee sample size used in calibrating test items and (b) the ratio of item bank size to test length. The results of the computer simulation study were clear: Both variables influenced the accuracy of test information functions, often substantially, with serious problems in accuracy arising when test items were calibrated on a modest examinee sample size (N = 500) and when test item banks were large in relation to test lengths.