LiME: Lightweight Mixture of Experts for Efficient Multimodal Multi-task Learning

A research team led by Md Kowsher and six other authors has introduced LiME (Lightweight Mixture of Experts), a novel approach that reimagines how mixture-of-experts architectures handle multimodal multi-task learning. Traditional MoE-PEFT methods require separate parameter-efficient fine-tuning adapters for each expert, causing trainable parameters to scale linearly with expert count and limiting architectural flexibility. LiME breaks this pattern by using a single shared PEFT module and modulating its output with lightweight expert vectors, dramatically reducing parameter requirements while generalizing to any PEFT method.

LiME introduces several technical innovations including zero-parameter routing that leverages existing frozen and adapted representations, eliminating the learned router parameters typically required per layer. The method also incorporates n-gram windowed routing and adaptive expert selection (Auto Top-K) based on routing confidence. The researchers provide theoretical proofs showing that more experts preserve more task-relevant information and that modulation approximates full expert-specific PEFT with bounded error.

Experiments on the challenging MMT-47 benchmark, which includes 47 tasks spanning text, image, and video modalities, demonstrate LiME's effectiveness. The approach achieves competitive or superior performance while using up to 4x fewer trainable parameters and enabling up to 29% faster training compared to traditional MoE-PEFT baselines. This represents a significant efficiency breakthrough for adapting large multimodal models to diverse downstream tasks.