Effect of Multiple Testing Adjustment in Differential Item Functioning Detection

In a typical differential item functioning (DIF) analysis, a significance test is conducted for each item. As a test consists of multiple items, such multiple testing may increase the possibility of making a Type I error at least once. The goal of this study was to investigate how to control a Type I error rate and power using adjustment procedures for multiple testing, which have been widely used in applied statistics. In the simulation, four distinct DIF methods were performed under various testing conditions. The methods were the Mantel–Haenszel (MH) method, the logistic regression (LR) procedure, the Differential Functioning Item and Test (DFIT) framework, and Lord’s chi-square test. As an adjustment procedure, the Bonferroni correction, Holm’s procedure, or the Benjamini and Hochberg (BH) false discovery rate was applied. The results showed the MH and the LR clearly benefited from Holm’s and the BH adjustments, whereas the DFIT and Lord’s chi-square test did not require adjustments for conditions under this study.