SeFoRA: Sketch-based federated LoRA beats state-of-the-art on GLUE
When clients use different LoRA ranks, aggregation breaks—SeFoRA fixes it with linear sketches.
Researchers propose SeFoRA, a sketch-aggregated federated LoRA algorithm for parameter-efficient fine-tuning of large neural networks. It lets clients use different LoRA ranks by transmitting a linear sketch of their local updates, enabling direct aggregation at the federator and alleviating the bilinear mismatch. A rank-homogeneous version, SeFoRA-Ho, is shown to converge to a neighborhood of the first-order stationary point at a rate of O(1/T). Experiments fine-tuning RoBERTa-Large on GLUE datasets show the algorithms outperform the state of the art.
- SeFoRA uses linear sketches to aggregate LoRA updates from clients with heterogeneous ranks, solving dimension mismatch and bilinear mismatch.
- Convergence to a first-order stationary point is proven at O(1/T) for the rank-homogeneous variant (SeFoRA-Ho).
- Outperforms state-of-the-art federated PEFT when fine-tuning RoBERTa-Large on GLUE benchmarks, with lower communication costs.
Why It Matters
Enables practical federated fine-tuning where each client chooses its own LoRA rank, removing a major coordination bottleneck.