Entrain framework cuts multimodal training variability by 10.6x

Training multimodal large language models (MLLMs) is notoriously difficult because datasets contain heterogeneous data from different modalities (text, images, audio) with independent variability. Existing approaches use dynamic model parallelism to adapt to changing workloads, but this adds overhead and complexity. In a new paper, researchers Insu Jang and Mosharaf Chowdhury from the University of Michigan propose Entrain, a framework that flips the assumption: instead of reacting to per-sample variability, Entrain profiles at the level of macroscopic batches. They prove that a single, fixed model-parallel configuration can achieve optimal load balancing when combined with a hierarchical microbatch assignment algorithm. This algorithm defers excess work within each iteration to stabilize variability across microbatches.

Experimental results show Entrain reduces workload variability across microbatches by up to 10.6× compared to baselines, and improves end-to-end training throughput by up to 1.40×. The key insight is that static parallelism, when paired with smart scheduling, can outperform dynamic approaches on multimodal workloads. This could simplify distributed training infrastructure for large AI models, reducing the need for complex runtime adaptation. The paper is available on arXiv and could influence how future MLLM training systems are designed.