Resampling-based multiple testing with asymptotic strong control of type I error

We define a general statistical framework for multiple hypothesis testing and show that the correct null distribution for the test statistics is obtained by projecting their true distribution onto the space of mean zero distributions. For common choices of test statistics (based on an asymptotically linear parameter estimator), this distribution is asymptotically multivariate normal with mean zero and the covariance of the vector influence curve for the parameter estimator. This test statistic null distribution can be estimated by applying the non-parametric or parametric bootstrap to correctly centered test statistics. We prove that this bootstrap estimated null distribution provides asymptotic strong control of most type I error rates. We show that obtaining a test statistic null distribution from a data null distribution only provides the correct test statistic null distribution if the covariance of the vector influence curve is the same under the data null distribution as under the true data distribution. This condition is the formal analogue of the subset pivotality condition (Westfall and Young (1993)). We also show that our multiple testing methodology controlling type I error is equivalent to constructing an error-specific confidence region for the true parameter values and checking if it contains the hypothesized value. We conclude with a discussion of applications.