Heterogeneous Federated Fine-Tuning with Parallel One-Rank Adaptation
New method solves key federated learning bottleneck, boosting accuracy by accommodating different client hardware.
Deep Dive
Researchers Zikai Zhang, Rui Hu, and Jiahao Xu propose Fed-PLoRA, a novel federated fine-tuning framework. It introduces Parallel One-Rank Adaptation (PLoRA), replacing multi-rank LoRA modules with parallel one-rank ones, and a Select-N-Fold strategy. This addresses noise from heterogeneous client resources (different LoRA ranks). Extensive experiments show Fed-PLoRA outperforms existing methods in accuracy and efficiency for LLM tasks, with code publicly available.
Why It Matters
Enables more practical, private, and performant collaborative AI training across phones, edge devices, and servers.