LLM Training Speed Boosted Up to 3.78x — This New Batching Method Comes With Hard Proof
Drop-in batch scheduler knows true sample cost after preprocessing—no cache needed.
Modern LLM training suffers from a fundamental mismatch: offline batch samplers decide batch composition before knowing each sample's true cost (after tokenization, augmentation, multimodal expansion). This wastes GPU memory and padding. A new paper from Dian Li et al. introduces Online Dynamic Batching (ODB), a drop-in DataLoader component that postpones batch formation until after preprocessing, when each sample's actual length is observable. ODB preserves DDP step alignment by formally solving the Distributed Group Alignment Problem (DGAP), proving deadlock-free bounded termination with default join-mode identity coverage and opt-in non-join sample-quota closure.
Benchmarked on public 2B/8B Qwen3-VL models over UltraChat, LLaVA, and ShareGPT4o, ODB achieves 1.58–2.51x literal emitted-sample throughput improvement on single-node Full FT/LoRA and 1.71–3.78x on two-node Full FT compared to fixed batching, with comparable quality. Against offline token-budget oracles (GMT/BMT), ODB stays within 15% on homogeneous datasets and actually outperforms them by 2.24–2.39x on high-variance ShareGPT4o. Production mixed-modality workloads reach 4.43x. ODB requires no model, optimizer, or kernel changes, and is released as an open-source package with lightweight trainer adapters.
- ODB boosts throughput 1.58–3.78x vs fixed batching on Qwen3-VL 2B/8B models
- Formally solves DGAP with deadlock-free termination guarantees
- Drop-in replacement—no model, optimizer, or kernel rewrites needed
Why It Matters
Cuts LLM fine-tuning costs significantly without quality loss, especially for mixed-modality or high-variance datasets.