MoP framework boosts MLLMs by 65% using privileged modalities during training
A new probing technique unlocks hidden value from training-only data, achieving up to 65% gains.
Multimodal LLMs typically assume all training modalities are available at inference, but real-world deployments often miss certain inputs (e.g., depth sensors, thermal cameras). A new paper from researchers across multiple institutions introduces Mixture of Probes (MoP), a framework designed for this "privileged modality" setting. MoP works by probing intermediate representations within a shared modality encoder rather than only final-layer alignment, allowing the model to extract both modality-specific and modality-general signals.
To prevent probe collapse and encourage genuine cross-modal learning, the authors also introduce MoP-X, a training strategy with a dedicated disentanglement loss. Evaluated on 8 tasks spanning 4 modalities (e.g., image, text, depth, audio), MoP achieves up to 65% relative improvement over strong baselines. This shows that auxiliary data, even when absent at inference, can be harnessed to significantly boost a model's core performance. Code and checkpoints will be released.
- MoP (Mixture of Probes) uses structured probing of intermediate encoder layers to separately capture modality-specific and shared representations.
- MoP-X training loss prevents probe collapse and maximizes cross-modal learning, enabling up to 65% performance gains.
- Evaluated across 8 tasks and 4 modalities, MoP consistently beats existing MLLM baselines in privileged modality settings.
Why It Matters
Enables MLLMs to leverage sensor-rich training data, boosting accuracy even when those sensors are absent at inference.