Per-Experiment Error Rates: The Hidden Costs of Several Multiple Comparison Procedures

For a number of common multiple comparison procedures controlling Type I error at .05 refers to what Ryan called the experimentwise error rate. This expression represents the chance of making at least one Type I error within a given experiment. This approach ignores, however, the potential detriments of multiple errors within a single experiment, that is, it fails to acknowledge what Ryan referred to as the error rate per experiment. The current study uses a computer simulation to evaluate the differences between experimentwise error rates and error rates per experiment for a variety of multiple comparison procedures. For pairwise comparisons, Newman-Keuls's and Tukey's tests are examined, as is Dunnett's test for comparisons with a control group and Scheffe's test for all possible post hoc comparisons. For planned contrasts, a standard Bonferroni and Shaffer's sequentially rejective Bonferroni are simulated using both a nonorthogonal and an orthogonal set.