Rollplex speeds up VLM post-training 1.3x with smarter GPU sharing

A team led by Hanfeng Lu at Tsinghua University, alongside 10 co-authors from multiple institutions, introduced Rollplex, a novel GPU runtime for Vision-Language Model (VLM) post-training via reinforcement learning. The core idea is to break the traditional serial execution of rollout, reference scoring, and actor training phases, which leaves GPUs underutilized when processing dense video inputs and long prompt prefixes. Rollplex overlaps prefix computation with rollout decoding, exploiting the fact that prefix processing is independent of the generated response. This spatial sharing is non-trivial: naive co-location of Qwen2.5-VL-32B would demand roughly 165 GiB HBM per GPU, and rollout vs. training prefer different tensor-parallelism degrees and weight layouts.

To overcome these constraints, Rollplex implements phase-aware memory management that controls HBM residency based on producer-consumer lifetimes and parallelism-aware weight sharing that reuses physical storage across compatible tensors, reconstructing only incompatible ones. On a 32-GPU H800 cluster, Rollplex achieves 1.23×–1.30× speedup over serial co-location and 1.57×–2.24× over disaggregated scheduling under the same GPU budget, all while maintaining synchronous on-policy semantics required for stable RL updates.