Federated fine-tuning unlocks private data for LLMs without sharing

A new cross-domain benchmark for federated fine-tuning of large language models (LLMs) shows that private, distributed data across institutions can be leveraged without compromising privacy. The framework, built on the OpenFL Federated Learning platform, allows nodes to jointly fine-tune a shared LLM while keeping raw data local. The study evaluates three parameter-efficient fine-tuning (PEFT) strategies—LoRA, QLoRA, and IA3—on four closed-ended QA and classification datasets: MedQA and MedMCQA (healthcare) and FPB and FiQA-SA (finance). The authors simulate non-IID data distributions to reflect real-world institutional heterogeneity.

Results demonstrate that federated fine-tuning achieves accuracy close to centralized training and significantly outperforms isolated single-institution learning. From a Green AI perspective, QLoRA and IA3 offer improved efficiency with only minimal accuracy degradation, making federated PEFT a viable path for adapting LLMs in regulated sectors. This breakthrough could unlock massive amounts of valuable private data—from hospital patient records to financial communications—for model improvement without violating privacy regulations.