Improving healthcare cost prediction for chronic disease through covariate clustering and subgroup analysis methods

Abstract Predicting healthcare costs for chronic diseases is challenging for actuaries, as these costs depend not only on traditional risk factors but also on patients’ self-perception and treatment behaviors. To address this complexity and the unobserved heterogeneity in cost data, we propose a dual-structured learning statistical framework that integrates covariate clustering into finite mixture of generalized linear models, effectively handling high-dimensional, sparse, and highly correlated covariates while capturing their effects on specific subgroups. Specifically, this framework is realized by imposing a penalty on the prior similarities among covariates, and we further propose an expectation-maximization-alternating direction method of multipliers (EM-ADMM) algorithm to address the complex optimization problem by combining EM with the ADMM. This paper validates the stability and effectiveness of the framework through simulation and empirical studies. The results show that our framework can leverage shared information among high-dimensional covariates to enhance fitting and prediction accuracy, while covariate clustering can also uncover the covariates’ network relationships, providing valuable insights into diabetic patients’ self-perception data.