叠加原理
变压器
计算机科学
工程类
数学
电气工程
数学分析
电压
作者
Ayoub Ben Chaliah,Hela Dellagi
出处
期刊:Qeios
[Qeios Ltd]
日期:2025-01-24
摘要
Catastrophic forgetting remains a major challenge when adapting large language models (LLMs) to new tasks or domains. Conventional fine-tuning often overwrites existing knowledge, causing performance degradation on original tasks. We introduce _Superposition in Transformers_, a novel architecture that leverages autoencoders to superimpose the hidden representations of a base model and a fine-tuned model within a shared parameter space. By using B-spline-based blending coefficients and autoencoders that adaptively reconstruct hidden states based on the input data distribution, our method effectively mitigates catastrophic forgetting and enables a new paradigm of “in-model” superposition. This approach preserves original model capabilities while allowing compact domain-specific expertise to be added, and it supports dynamic switching between model states during inference.
科研通智能强力驱动
Strongly Powered by AbleSci AI