Bayesian adaptive design for clinical trials with potential subgroup effects

Adaptive clinical trial designs increasingly aim to improve efficiency while accommodating subgroup heterogeneity, yet most existing methods fix assumptions about drug efficacy and subgroup effects. We propose a Bayesian adaptive design that explicitly models and learns from uncertainty in both components. A hierarchical mixture prior represents uncertainty about overall treatment efficacy and the magnitude of a biomarker-defined subgroup effect. Interim data are used to update these hyperparameters into posterior distributions, enabling a decision-theoretic framework that adaptively selects the optimal testing strategy among three options: continuing with the overall population, focusing on the subgroup, or conducting a joint test of both. When joint testing is chosen, the posterior information further determines the optimal allocation of Type I error between populations by selecting an evidence-based α -splitting parameter that maximizes expected power under error-rate constraints. The resulting optimization is solved efficiently using GPU-accelerated quasi-Monte Carlo integration and smooth search procedures. Simulation studies across a range of subgroup prevalences and effect sizes demonstrate that the proposed design maintains nominal error control, achieves superior power and decision accuracy, and adapts appropriately to prior misspecification. By unifying posterior learning and adaptive α -allocation within a principled Bayesian framework, this design provides a transparent and computationally practical tool for confirmatory clinical trials with uncertain subgroup effects, supporting precision-medicine decision-making and regulatory reproducibility.