Identifying spatially variable genes by projecting to morphologically relevant curves

Abstract Spatial transcriptomics enables high-resolution gene expression measurements while preserving the two-dimensional spatial organization of the sample. A common objective in spatial transcriptomics data analysis is to identify spatially variable genes within predefined cell types or regions within the tissue. However, these regions are often implicitly one-dimensional, making standard two-dimensional coordinate-based methods less effective as they overlook the underlying tissue organization. Here we introduce a methodology grounded in spectral graph theory to elucidate a one-dimensional curve that effectively approximates the spatial coordinates of the examined sample. This curve is then used to establish a new coordinate system that better reflects tissue morphology. We then develop a generalized additive model (GAM) to detect genes with variable expression in the new morphologically relevant coordinate system. Our approach directly models gene counts, thereby eliminating the need for normalization or transformations to satisfy normality assumptions. We demonstrate improved performance relative to current methods based on hypothesis testing, while also accurately estimating gene expression patterns and precisely identifying spatial loci where deviations from constant expression are observed. We validate our approach through extensive simulations and by analyzing experimental data from multiple platforms such as Slide-seq and MERFISH.