HESpotEx: a dual-stream deep learning framework for spot-level gene expression prediction from histological images

Whole-slide histopathological images (WSIs) constitute a fundamental approach in disease diagnosis and prognosis. Recently emerging spatial transcriptomics (ST) methods can reveal the spatial gene expression landscape behind the histopathological images, but with much higher cost. Here, therefore, we propose HESpotEx, a dual-stream multimodal deep learning framework to predict the spatial gene expression patterns solely from WSI images. Leveraging graph attention autoencoders, an image encoder and a graph convolution network decoder, HESpotEx is capable of predicting expressions of up to 5,457 genes across individual spatial sampling spots from WSIs. HESpotEx exhibits superior performance and better robustness on ST datasets from various cancer and noncancer samples as well as on a large-scale The Cancer Genome Atlas WSI dataset. Moreover, on our in-house WSI dataset, HESpotEx also underscores diagnosis-associated WSI patches. Finally, HESpotEx shows better cross-sectional consistency in the latest high-resolution ST datasets. Together, our results demonstrate the potential of HESpotEx to decipher the spatial molecular characteristics underlying tissue histological patterns.