Online rules for control of false discovery rate and false discovery exceedance

Multiple hypothesis testing is a core problem in statistical inference and arises in almost every scientific field. Given a set of null hypotheses $\mathcal{H}(n)=(H_{1},\ldots,H_{n})$, Benjamini and Hochberg [J. R. Stat. Soc. Ser. B. Stat. Methodol. 57 (1995) 289–300] introduced the false discovery rate ($\mathrm{FDR}$), which is the expected proportion of false positives among rejected null hypotheses, and proposed a testing procedure that controls $\mathrm{FDR}$ below a pre-assigned significance level. Nowadays $\mathrm{FDR}$ is the criterion of choice for large-scale multiple hypothesis testing. In this paper we consider the problem of controlling $\mathrm{FDR}$ in an online manner. Concretely, we consider an ordered—possibly infinite—sequence of null hypotheses $\mathcal{H}=(H_{1},H_{2},H_{3},\ldots)$ where, at each step $i$, the statistician must decide whether to reject hypothesis $H_{i}$ having access only to the previous decisions. This model was introduced by Foster and Stine [J. R. Stat. Soc. Ser. B. Stat. Methodol. 70 (2008) 429–444]. We study a class of generalized alpha investing procedures, first introduced by Aharoni and Rosset [J. R. Stat. Soc. Ser. B. Stat. Methodol. 76 (2014) 771–794]. We prove that any rule in this class controls online $\mathrm{FDR}$, provided $p$-values corresponding to true nulls are independent from the other $p$-values. Earlier work only established $\mathrm{mFDR}$ control. Next, we obtain conditions under which generalized alpha investing controls $\mathrm{FDR}$ in the presence of general $p$-values dependencies. We also develop a modified set of procedures that allow to control the false discovery exceedance (the tail of the proportion of false discoveries). Finally, we evaluate the performance of online procedures on both synthetic and real data, comparing them with offline approaches, such as adaptive Benjamini–Hochberg.