EpiAgent: Foundation Model for Single-Cell Epigenomics

Large-scale foundation models have recently opened a new avenue to artificial general intelligence for life sciences, showing great promise in the analysis of single-cell transcriptomic data. Nevertheless, such challenges as the tremendous number of signaling regions, extreme data sparsity, and the nearly binary nature of single-cell epigenomic data have prevented the construction of a foundation model for epigenomics thus far, though it is evident that abundant epigenomic properties such as chromatin accessibility provide more decisive insights into cell states than transcriptomics, shaping the chromatin regulatory landscapes that control transcription in distinct cell types. Here, we introduce EpiAgent, the first foundation model for single-cell chromatin accessibility data, pretrained on a manually curated large-scale Human-scATAC-Corpus that is comprised of approximately 5 million cells and 35 billion tokens. EpiAgent encodes chromatin accessibility patterns of cells as concise “cell sentences,” and employs a bidirectional attention mechanism to capture cellular heterogeneity behind regulatory networks. With comprehensive benchmarks, we demonstrate that EpiAgent excels in typical downstream tasks, including unsupervised feature extraction, supervised cell type annotation, and data imputation. By incorporating external embeddings, EpiAgent facilitates the prediction of cellular responses to both out-of-sample stimulated and unseen genetic perturbations, as well as reference data integration and query data mapping. By simulating the knockout of key cis-regulatory elements, EpiAgent enables in-silico treatment for cancer analysis. We further extended zero-shot capabilities of EpiAgent, allowing direct cell type annotation on newly sequenced datasets without additional training.