Transformer-Based Topic Modeling for Research Literature

Scalable techniques to arrange, condense, and evaluate textual data are required due to the scientific literature’s exponential growth. In this work, a reproducible pipeline based on BERTopic is presented, which combines class-based TF-IDF for topic labelling, UMAP for dimensionality reduction, HDBSCAN for density-aware clustering, and transformer-based sentence embeddings. Compared to traditional baselines like LDA and TF–IDF+KMeans, the pipeline produces comprehensible topics, strong clustering metrics, and interpretable labels when tested on a multi-domain corpus of roughly 10,462 arXiv abstracts from 2010 to 2023. We offer rich visualizations (heatmap, dendrogram, distribution charts), quantify improvements (silhouette and coherence), and talk about design decisions and constraints for real-world implementation.