AMDP: New pipeline parallelism speeds large model training with stable convergence

Existing asynchronous pipeline parallelism for training large models often suffers from convergence degradation due to parameter mismatch between forward and backward passes. To address this, researchers from an undisclosed institution introduced AMDP (Asynchronous Multi-Directional Pipeline Parallelism). The method limits each pipeline's first stage to processing at most two minibatches before backpropagation, thereby bounding the number of parameter updates that occur between forward and backward passes. This reduces the staleness of gradients that can destabilize training. To compensate for the resulting pipeline bubbles, AMDP spawns multiple concurrent pipelines and dynamically adjusts their count based on pipeline depth, maintaining high hardware utilization.

AMDP further accumulates gradients over several minibatches and applies them in a single optimization step, ensuring that only a bounded number of minibatches ever experience parameter mismatch—and that mismatch is limited to within one optimization step. Experiments on GPT-style (e.g., GPT-2, GPT-3 scale) and BERT-style models demonstrate that AMDP significantly accelerates training throughput while preserving model convergence. The paper, accepted at ICML 2026, provides detailed performance results across various model sizes and cluster configurations. This technique offers a practical path for organizations training ever-larger language models to reduce wall-clock time without sacrificing model quality.