Abstract 1138: Decoding tumor microenvironment with deep learning: merging spatial transcriptomics and histopathology

Abstract Spatial transcriptomics (ST) provides unprecedented insights into the spatial landscape of gene expression across tissues, crucial for understanding cancer heterogeneity. Many ST analysis methods, however, do not fully utilize the rich morphological information in histopathology images and are limited by small sample sizes and high costs. We developed an innovative multimodal framework combining spatial transcriptomics with histopathological features using advanced image-aware deep-learning models, aimed at identifying significant patterns missed by traditional clustering and leveraging these models to impute gene expression across tissue slides for large-scale biomarker discovery and mechanistic studies. Our study focuses on glioblastoma, triple-negative breast cancer, and colorectal cancer, utilizing fully sequenced and clinically characterized samples from 10x Genomics Visium ST and Visium HD, along with 10x to 40x high resolution whole-slide histopathology images. We segmented ST histological images into tiles; for Visium ST, tiles were squares with widths equivalent to spot diameters, while for Visium HD, tiles corresponded to 16-micron bins. Using models like ResNet and Vision Transformer (ViT), we extracted morphological features, creating a detailed feature matrix per spot. Unsupervised clustering applied to morphological features to explore underlying patterns without prior labels. Integrative analysis merged gene expression with morphological data to identify biologically significant clusters. Supervised learning predicted tumor microenvironment characteristics: local effects like hypoxia, specific cell-types such as microglia, and gene-level expressions. A classifier was trained to predict tile-level cell types and processors, leveraging expression levels of pertinent marker gene sets and gene signatures for each tumor type. Additionally, this model predicted tile-level gene expression by initially selecting genes with the highest Moran’s I values, which are indicative of spatial heterogeneity. This method assessed the predictive accuracy of various pre-trained model features. Findings demonstrate that ViT models achieved high Adjusted Rand Index scores, enhancing the identification of biologically enriched clusters as validated by gene set enrichment analysis. ViT predicted specific biological characteristics across slides, demonstrating its potential for imputing gene expression signatures in extensive pathology datasets. This study underscores the potential of integrating spatial transcriptomics with deep-learning, especially Vision Transformers, enhancing our ability to analyze tumor microenvironments for biomarker discovery and deeper biological insights. Citation Format: Jiarong Song, John J. Lee, Rayyan Aburajab, Bohan Zhang, Kayla Xu, Yonatan Amzaleg, Jian Ye, Rania Bassiouni, John Carpten, David Craig. Decoding tumor microenvironment with deep learning: merging spatial transcriptomics and histopathology [abstract]. In: Proceedings of the American Association for Cancer Research Annual Meeting 2025; Part 1 (Regular Abstracts); 2025 Apr 25-30; Chicago, IL. Philadelphia (PA): AACR; Cancer Res 2025;85(8_Suppl_1):Abstract nr 1138