A Step Toward Federated Pretraining of Multimodal Large Language Models
New method trains vision-language models on private data without centralizing it, tackling key aggregation challenges.
A team of researchers has introduced Fed-CMP, a pioneering framework designed to enable the federated pretraining of Multimodal Large Language Models (MLLMs). The core challenge they address is the saturation of public training data; while vast troves of private, multimodal data exist in hospitals, companies, and personal devices, privacy concerns prevent its centralization. Federated Learning (FL) offers a solution by training models across decentralized data silos, but prior work focused only on fine-tuning existing models. Fed-CMP tackles the more complex problem of foundational pretraining, specifically targeting the 'alignment' phase where a model learns to connect visual features from an encoder with a text-based LLM.
The Fed-CMP framework is architecturally lightweight, keeping the vision encoder and the large language model frozen. It only trains the 'cross-modal projector'—the component that translates between vision and language representations—in a collaborative, federated manner. The researchers identified two major technical hurdles in this setting: parameter interference when aggregating locally trained projectors from different clients, and unstable gradient oscillations during training. To solve these, Fed-CMP employs two novel techniques: Canonical Reliability-Aware Aggregation, which decomposes client models into shared and client-specific parts for cleaner fusion, and Orthogonality-Preserved Momentum, which stabilizes the optimization process.
Extensive experiments across four constructed federated pretraining scenarios, using public datasets to simulate private data distributions, demonstrated that Fed-CMP significantly outperforms existing federated learning baselines. This work, formalized as the 'Federated MLLM Alignment (Fed-MA)' task, represents a crucial step toward building powerful, general-purpose multimodal AI without compromising data privacy, potentially unlocking petabytes of currently inaccessible training data.
- Proposes Fed-CMP for federated pretraining of MLLMs, focusing on aligning vision and language encoders without centralizing private data.
- Introduces Canonical Reliability-Aware Aggregation to mitigate parameter interference and Orthogonality-Preserved Momentum to stabilize gradient optimization.
- Validated across four federated scenarios, showing significant performance gains over existing methods, enabling training on vast, siloed datasets.
Why It Matters
Unlocks training on massive private image-text datasets (medical, corporate) for building better multimodal AI, without violating data privacy regulations.