LoRA fusion fails cross-domain tasks, researchers propose 3.8% improvement fix

A new study from researchers including Yatong Wang tackles a fundamental challenge in cloud-edge AI: how to combine knowledge from multiple private edge devices into a cloud-hosted large language model (LLM) without uploading sensitive data. The typical approach—training LoRA adapters locally and fusing them in the cloud—assumes edge devices can run full-scale LLMs, which is unrealistic. The team proposes a prune-train-recover framework that first prunes the cloud LLM to fit on edge devices, trains LoRA adapters locally, then recovers the full model in the cloud for fusion. This enables privacy-preserving adaptation across domains.

To evaluate cross-domain capabilities, the authors introduce MMLU-CD, a benchmark composing multiple domain samples into single test instances. Their empirical results are stark: existing LoRA fusion methods often perform worse than the base LLM on cross-domain tasks, revealing a critical oversight in current pipelines. They attribute this failure to parameter conflicts among adapters and propose LoRA-CR, a simple conflict-resolution module that selectively mitigates conflicting updates. LoRA-CR improves fusion performance by up to 3.8%, but the researchers emphasize that conflict mitigation remains an open problem warranting further investigation.