计算机科学
联合学习
分布式数据库
分布式学习
基于知识的系统
万维网
移动计算
知识管理
知识库
计算机网络
服务器
知识获取
知识抽取
移动设备
知识工程
分布式计算
钥匙(锁)
领域知识
知识表示与推理
分布式知识
作者
Wei Sun,Xianda Wang,Zhicheng Liang,Tianyi Gong,Wanshun Lan,Yingchun Chen,Haoyue Li,Fangxin Wang
标识
DOI:10.1109/tmc.2026.3674545
摘要
Federated learning (FL) stands out as a promising solution to address the ever-increasing data scarcity problem of large language model (LLM) training through collecting data from distributed sources. The unique challenges however exist in three aspects, the extreme large communication overhead due to frequent model aggregation, the suboptimal performance due to data heterogeneity, and privacy leakage risk due to model parameter exchange. To address these issues, we propose Federated Knowledge Learning (FKL), a novel token-level knowledge sharing framework tailored for LLM to achieve effective task relevant knowledge aggregation. FKL offers the following advantages: (1) Adaptive Flexibility: Clients and servers can employ models customized for their specific tasks while ensuring seamless collaboration. Each node within the system functions independently and can be flexibly adapted as needed. (2) Communication Efficiency: Instead of huge model or parameter exchange, the unique knowledge sharing mechanism of FKL can significantly reduce the communication overhead while delivering superior performance. (3) Privacy Safeguards: The distributed token-level knowledge-sharing mechanism ensures privacy by deidentifying sensitive information and mitigating potential leakage risks. Experiments on LLMs (125M-7B parameters) validate FKL's versatility, achieving 50% higher accuracy on GLUE benchmarks and a notable 0.7x improvement in privacy task response quality compared to existing FL-based LLM approaches.
科研通智能强力驱动
Strongly Powered by AbleSci AI