Power and Type I Error Control for Univariate Comparisons in Multivariate Two-Group Designs

Simulations were conducted to evaluate the statistical power and Type I error control provided by several multiple-comparisons procedures in two-group designs. Stepwise Bonferroni-based procedures, which are known to control the familywise Type I error rate, tended to be more powerful than other methods but did not control the per-family Type I error rate (PFER). It is proposed that more attention should be given to the PFER, particularly with regard to these procedures. Only two methods controlled the PFER: the classical Bonferroni procedure and a modified version of MANOVA-protection. Which of these two procedures was more powerful depended on multiple factors that this article describes in detail and illustrates graphically. It is concluded that which multiple-comparisons procedure is preferable depends on the number of outcome variables, the importance of the PFER, the necessity of confidence intervals, and the extent to which significance in multiple variables is more valuable than significance in one variable.