Hunting for Significance With the False Discovery Rate

When testing a single hypothesis, it is common knowledge that increasing the sample size after nonsignificant results and repeating the hypothesis test several times at unadjusted critical levels inflates the overall Type I error rate severely. In contrast, if a large number of hypotheses are tested controlling the False Discovery Rate, such “hunting for significance” has asymptotically no impact on the error rate. More specifically, if the sample size is increased for all hypotheses simultaneously and only the test at the final interim analysis determines which hypotheses are rejected, a data dependent increase of sample size does not affect the False Discovery Rate. This holds asymptotically (for an increasing number of hypotheses) for all scenarios but the global null hypothesis where all hypotheses are true. To control the False Discovery Rate also under the global null hypothesis, we consider stopping rules where stopping before a predefined maximum sample size is reached is possible only if sufficiently many null hypotheses can be rejected. The procedure is illustrated with several datasets from microarray experiments.