CLAIR enables safe federated LoRA fine-tuning for LLMs

Low-rank adaptation (LoRA) has become a go-to method for parameter-efficient fine-tuning of large language models. But when multiple clients collaborate under a federated setting—each holding private data—the challenge of handling heterogeneous, potentially contaminated clients emerges. Researchers from the field propose CLAIR (Collaborative Low-rank Alignment and Identifiable Recovery), a framework that tackles this exact problem. CLAIR assumes clients share only partial LoRA structure and that a substantial subset may be poisoned. It works by decomposing local matrix-valued updates into a shared low-rank component plus a block-sparse contamination term, enabling both subspace recovery and client-level anomaly detection. The approach is broadly applicable to linear regression, neural networks, and LLM modules.

On the theoretical side, the authors prove exact recovery of the shared LoRA subspace in the noiseless case and stable recovery when preliminary estimation errors exist. They also derive conditions for consistent identification of the collaborative client set. The key insight: cross-client averaging within the recovered subspace reduces off-subspace error, while preserving client-specific variation—improving over local fine-tuning as long as the oracle gains outweigh subspace estimation costs and benign heterogeneity. Empirically, feeding a Transformer on a text-copying task, CLAIR outperformed both local-only fine-tuning and non-robust federated averaging, accurately flagging contaminated clients and boosting benign-client performance. This work opens a practical path for privacy-preserving, collaborative LLM customization that is resilient to data poisoning.