FedALoRA: Adaptive Local LoRA Aggregation for Personalized Federated Learning in LLM

Federated Large Language Model (FedLLM) shows excellent potential in collaboratively training large language models (LLM) under the federated learning (FL) framework, which is benefiting from its privacy protection advantage. However, FedLLM faces the significant challenge of the non-IID problem. In the real world, there are often cross-source or even cross-domain language set data between IoT devices. To address the issue, we propose a new FedLLM framework FedALoRA via personalized and efficient parameter fine-tuning (PEFT). Specifically, the proposed scheme combines the personalized aggregation method and the LoRA method, which can adaptively aggregate the downloaded global model and local model to the local target on each client while ensuring low training costs. This adaptation initializes the local model before each iterative training, enabling clients to learn general knowledge while enhancing their understanding of their own domain knowledge. Extensive experiments and analysis on cross-domain non-IID settings and the financial datasets on Dirichlet non-IID settings demonstrate the effectiveness and superiority of FedALoRA.