LightCrafter uses PBR and diffusion for consistent video relighting
New hybrid pipeline outperforms prior art with physically-based proxy videos
Video relighting has long struggled with balancing temporal consistency and physically accurate light transport. Existing methods fall into two camps: inverse-rendering pipelines that reconstruct geometry and materials but suffer from noisy reconstructions and miss global illumination effects, and generative video-to-video translation approaches that struggle with long-form stability and limited lighting control. LightCrafter, developed by researchers including Zixin Guo and Deva Ramanan, introduces a clever hybrid that bridges both worlds.
The core idea is to reformulate video relighting as a translation of a proxy video. Instead of directly translating the input to a target, they first produce a physically-based rendering (PBR) of the input under the desired illumination. This PBR proxy bakes the lighting target into the pixels, removing the need for the diffusion model to learn abstract concepts like environment maps. Then, a post-trained CogVideoX diffusion model refines the proxy to capture effects like global illumination that PBR alone misses. The result is controllable, temporally consistent relighting that outperforms prior art on real-world benchmarks. The team will release their synthetic benchmark, metrics, and code.
- Uses a PBR proxy video under target illumination to condition the diffusion model, avoiding direct lighting concepts
- Post-trains CogVideoX on synthetic video pairs and real-world unpaired videos for better global illumination effects
- Outperforms prior state-of-the-art on real-world relighting benchmarks; synthetic benchmark and code to be released
Why It Matters
Enables filmmakers and VFX artists to relight video footage with stability and physical accuracy, replacing manual work.