FlashCP cuts LLM training communication by 1.63x with load-balanced context parallelism

Training large language models with long contexts (e.g., 128K+ tokens) requires context parallelism (CP) to split sequences across GPUs and reduce memory pressure. But current CP methods suffer from workload imbalance, inefficient kernels, and redundant communication—because they shard sequences statically and communicate entire key-value (KV) tensors between devices. FlashCP, introduced by Zheng Wang and seven co-authors, tackles all three issues in a single load-balanced, communication-efficient framework.

FlashCP's key innovations are two-fold. First, it uses a sharding-aware communication mechanism that eliminates redundant KV tensor transfers by only sending the necessary slices. Second, it proposes Whole-Doc sharding—a strategy that groups entire documents rather than arbitrary chunks—to maximize communication savings while keeping workloads balanced across GPUs. Because different datasets benefit from different sharding plans, FlashCP also includes a heuristic search algorithm to find near-optimal plans that combine Whole-Doc and Per-Doc sharding. In experiments, FlashCP achieves up to 1.63x speedup over existing CP frameworks across diverse long-context datasets, making it a practical drop-in improvement for large-scale LLM training.