计算机科学
一致性(知识库)
前缀
秩(图论)
基础(线性代数)
基质(化学分析)
数据挖掘
理论计算机科学
适应(眼睛)
数据聚合器
空格(标点符号)
分布式计算
数据一致性
矩阵分解
备份
方案(数学)
期限(时间)
分解
服务(商务)
情报检索
作者
Pengpeng Qiao,Yang Cao,Lingling Zhang,Guo Cheng,Junwei Chen,Manjiang Yu,Wei Yang Bryan Lim,Masatoshi Yoshikawa
标识
DOI:10.48550/arxiv.2611.07598
摘要
Federated fine-tuning of large language models (LLMs) under heterogeneous client capacities is increasingly important, and low-rank adaptation (LoRA) with heterogeneous ranks makes it practical. Existing server-side aggregators compose local matrix updates $ΔW_i$ from client-specific LoRA factors $(A_i,B_i)$ before aggregating them. Although these matrix updates lie in the same weight space $\mathbb{R}^{m\times n}$ and can be averaged directly, the LoRA factors that generate them may expressed client-specific local basis. In the typical compose--aggregate--factorize--dispatch loop, this basis inconsistency can make the rank-$r_i$ reference prefix dispatched to lower-rank clients unstable after server-side refactorization. In this work, we propose \textbf{FedAbA}, a basis-consistent federated LoRA aggregation framework for heterogeneous ranks. FedAbA first extracts each client's local basis, aligns it to the corresponding prefix of the global reference basis, and constructs a basis-consistent reconstruction of each local matrix update before aggregation. It then aggregates these reconstructions with weights that combine client data size and consistency score, followed by exact coefficient-space SVD refactorization and gauge fixing before rank-prefix dispatch. Our theoretical analysis clarifies the role of basis consistency and supports the proposed consistency-aware aggregation mechanism. Extensive experiments show that FedAbA outperforms representative federated LoRA baselines, with the largest gains under the strongest rank heterogeneity.
科研通智能强力驱动
Strongly Powered by AbleSci AI