GeneMamba: Efficient and Effective Foundation Model on Single Cell Data

Transformer-based models have achieved strong performance in single-cell transcriptomics, yet scaling them to genome-wide gene sequences remains challenging due to the quadratic cost of self-attention. Recent state space models (SSMs) offer linear-time sequence modeling, but unidirectional designs may introduce ordering biases and overlook biological structure, potentially limiting robustness across datasets. To address these challenges, we propose GeneMamba, a bidirectional state space model tailored for single-cell RNA sequencing. GeneMamba integrates three key components: (1) Efficiency: BiMamba blocks enable bidirectional processing with O(L) complexity, providing substantial speedup over transformer baselines while maintaining linear scaling; (2) Biological integration: a pathway-aware contrastive objective incorporates curated pathway information to guide representation learning; (3) Scalability: large-scale pretraining on approximately 30 million curated cells (from ~50 million raw CELLXGENE profiles) supports transfer across multiple downstream tasks without task-specific architectures. Across benchmarks including multi-batch integration, cell type annotation, and gene rank reconstruction, GeneMamba demonstrates competitive performance relative to strong baselines such as scGPT and Geneformer, with improvements observed in several settings. Ablation studies further suggest that bidirectional modeling and pathway-aware pretraining contribute to these gains. Overall, GeneMamba provides an efficient and biologically informed framework for foundation modeling in single-cell transcriptomics.