The Multimodal Pretraining Framework CarHE Predicts Spatial Transcriptomics in Tumors from Routine Pathology Images

Abstract Spatial transcriptomic analyses provide spatially resolved gene expression data that can provide insights into complex biological processes. However, current spatial transcriptomics approaches remain financially prohibitive and restricted in resolution, scalability, and gene coverage, limiting broader adoption for large-scale studies. Here, we developed CarHE (contrastive alignment of gene expression for hematoxylin and eosin images), a multimodal pretraining framework that infers high-dimensional spatial transcriptomic profiles from routine H&E-stained slides. By using contrastive learning to align cell type–specific transcriptomic information with histological features, CarHE achieved high prediction accuracy across evaluated datasets and spatial transcriptomics platforms. CarHE approximated spatially organized pathological microenvironment features consistent with tertiary lymphoid structure (TLS)-associated regions in breast cancer, lung cancer, melanoma, and clear cell renal cell carcinoma. Additionally, CarHE inferred approximated 3D spatial transcriptomic context from 2D images, providing more informative neighborhood context than 2D visualization. In a cohort of 880 lung cancer patients, CarHE-derived features were associated with disease-free survival and outperformed current approaches. Overall, CarHE provides a cost-effective and scalable framework for H&E-based spatial inference, supporting further validation toward translational research applications.