Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity

Researchers Yucheng Zhou and Jianbing Shen have developed a novel training acceleration method for autoregressive video generation models, addressing a critical bottleneck in AI video synthesis. Their paper, "Accelerating Training of Autoregressive Video Generation Models via Local Optimization with Representation Continuity," tackles the prohibitive computational cost and prolonged training times that constrain models like OpenAI's Sora. The core problem they identified is that simply training on fewer video frames to save time exacerbates error accumulation and creates jarring inconsistencies in the final generated sequences.

To solve this, the team introduced a two-pronged approach. First, the Local Optimization (Local Opt.) method optimizes tokens within specific, localized windows while still leveraging broader contextual information, which significantly reduces the propagation of errors through the video timeline. Second, inspired by mathematical Lipschitz continuity, they developed a Representation Continuity (ReCo) strategy. ReCo applies a continuity loss function to constrain how much the model's internal representations can change between frames, improving overall robustness and temporal coherence.

Extensive experiments on both class-conditional and text-to-video datasets demonstrate that this combined Local Opt + ReCo framework achieves performance superior to standard baseline training methods. Most impressively, it accomplishes this while cutting the total training cost in half, all without sacrificing the quality of the generated videos. This represents a major efficiency breakthrough for a field where training runs can consume millions of dollars in compute resources and weeks of time.