Re-evaluating the role of pilot trials in informing effect and sample size estimates for full-scale trials: a meta-epidemiological study

Background Some have argued that pilot trials have little value for informing the expected effect size of a subsequent large trial. This study aims to empirically evaluate the roles of pilot trials in informing the effect and sample size estimates of a full-scale trial. Methods We conducted a search in PubMed on 19 February 2022, for all pilot trials published between 2005 and 2018 and their subsequent full-scale trials. We analysed the agreement in results by comparing the direction and magnitude of the effect size in the pilot trial and full-scale trial. Logistic regression was used to explore whether a significant pilot trial and other characteristics were associated with a significant full-scale trial. Results A total of 248 pairs of pilot and full-scale trials were analysed. Full-scale trials with a significant pilot trial were 2.72 times more likely to find a significant result for the primary efficacy outcome than those with a non-significant pilot trial (95% CI 1.52 to 4.86, p=0.001). The association remained significant irrespective of changes made to the trial design. In 73% of the pairs, the pilot trial produced a larger point estimate than the subsequent full-scale trial, but 87% of pairs had a 95% CI estimated by the pilot trial that covered the full-scale trial point estimate. Full-scale trials with a sample size estimated using the SD from the pilot trial were less likely to yield a significant result (OR=0.26, 95% CI 0.10 to 0.65, p=0.004). Conclusion Pilot trials can provide strong signals on intervention efficacy. When determining the sample size for full-scale trials, using the CI bounds from the pilot trials instead of the point estimate may improve power estimation.