ReGenVC squeezes talking-head video to 26kB, decoding live at 24fps

ReGenVC, introduced by Zheyuan Zhang and Johnson Wu in arXiv:2607.28144, is the first end-to-end generative video codec to deliver ultra-low-bitrate encoding alongside real-time decoding. The encoder compresses talking-head video into a compact bitstream of roughly 26kB per 77-frame sequence, using a neurally compressed first frame, per-frame pose keypoints, and metadata. Compared to traditional codecs like x264 and x265, which need 250–280kB for artifact-free output on the same content, ReGenVC achieves a 10x bitrate reduction. At matching ultra-low bitrates, conventional codecs degrade into blocking artifacts while ReGenVC stays sharp, leveraging a strong generative prior from its diffusion transformer backbone.

Real-time decoding was the key obstacle: multi-step sampling with transformer and VAE components was too slow for interactive use. The authors solved this with four-step distillation plus three system-level optimizations: eight-GPU sequence parallelism using Ulysses and Ring, a spatially-split VAE, and a three-stage overlapped pipeline. On an 8-GPU node, ReGenVC sustains 24fps output, hitting a 972ms latency per 25-frame window—within the 1000ms budget for live streaming. A hybrid CPU-GPU deployment runs the encoder at 24fps on CPU while offloading one-shot conditioning encoders to CPU, cutting per-GPU memory from 21.1GB to 7.7GB. This makes ReGenVC practical for browser-based live video, potentially reshaping video conferencing and streaming at extreme low bitrates.