可解释性
计算机科学
人工智能
自然语言处理
语义学(计算机科学)
钥匙(锁)
简单(哲学)
语言理解
机器学习
特征(语言学)
语义特征
语言模型
自然语言
词汇
计算
语义属性
语义记忆
语法
计算语言学
深度学习
作者
Usha Bhalla,Alex Oesterling,Claudio Mayrink Verdun,Himabindu Lakkaraju,Flávio P. Calmon
标识
DOI:10.48550/arxiv.2511.05541
摘要
Translating the internal representations and computations of models into concepts that humans can understand is a key goal of interpretability. While recent dictionary learning methods such as Sparse Autoencoders (SAEs) provide a promising route to discover human-interpretable features, they often only recover token-specific, noisy, or highly local concepts. We argue that this limitation stems from neglecting the temporal structure of language, where semantic content typically evolves smoothly over sequences. Building on this insight, we introduce Temporal Sparse Autoencoders (T-SAEs), which incorporate a novel contrastive loss encouraging consistent activations of high-level features over adjacent tokens. This simple yet powerful modification enables SAEs to disentangle semantic from syntactic features in a self-supervised manner. Across multiple datasets and models, T-SAEs recover smoother, more coherent semantic concepts without sacrificing reconstruction quality. Strikingly, they exhibit clear semantic structure despite being trained without explicit semantic signal, offering a new pathway for unsupervised interpretability in language models.
科研通智能强力驱动
Strongly Powered by AbleSci AI