SierpinskiCam uses fractal patterns for camera-controlled video retaking

Generating novel views of a scene along arbitrary camera trajectories from a single video—dubbed video retaking—is a major challenge for content creation and VFX. Existing geometry-guided methods reconstruct a 4D representation from the source video, then render it along the target trajectory to condition diffusion models. But as the virtual camera moves far from the original path, newly revealed regions become sparse or missing entirely. SierpinskiCam, developed by Suttisak Wizadwongsa, Hyelin Nam, Supasorn Suwajanakorn, and Jeong Joon Park at KAIST, overcomes this by augmenting geometry cues with a Sierpinski dome texture. The fractal pattern provides rich, trackable features even under extreme viewpoint changes, keeping the diffusion model grounded.

The team also introduces a reference video conditioning mechanism that appends source-video tokens to the target-token sequence, separating the two streams with negative RoPE (rotary position embedding) indices. This lets the model reference the original appearance without architectural modifications or per-video tuning. Extensive experiments across diverse and challenging retaking scenarios show SierpinskiCam achieves significant gains in camera controllability, geometric consistency, and video quality over prior methods. The paper, 20 pages with 13 figures, is available on arXiv and includes a project page for code and demos.