LooseControlVideo’s 3D Boxes Make Multi-Object Video Generation Shockingly Simple
Sparse 3D blocking gives up to 3x better trajectory control over existing methods.
LooseControlVideo introduces a novel approach to spatial orchestration in text-to-video generation. Instead of requiring dense, frame-accurate depth maps, users author high-level motion with sparse, oriented 3D boxes that act as a 'blocking' proxy. The system is built by fine-tuning a Wan 2.2 diffusion backbone on datasets annotated with DNOCS — a custom encoding that captures 3D size, orientation, and depth-ordered occlusions. This allows the model to infer realistic occlusions, interactions, and deformable object dynamics from loose spatial hints. Users can also perform localized refinements (e.g., adjusting a jump trajectory) without disrupting the global scene context.
Extensive evaluations on nuScenes, HO-3D, and BEHAVE benchmarks show that LooseControlVideo significantly outperforms existing 2D-box and flow-based baselines. Key metrics include a 1.2× to 3× improvement in Trajectory Error, a 2× gain in Rigid Motion Consistency, and a 1.5× to 2× increase in Occlusion Accuracy. The work demonstrates that oriented 3D primitives provide a powerful geometric prior for complex, multi-agent video authoring — potentially enabling filmmakers and content creators to direct AI-generated videos with far less manual effort.
- Uses sparse 3D oriented boxes (blocking) instead of dense frame-by-frame depth guidance.
- Fine-tuned on Wan 2.2 with a new DNOCS encoding for size, orientation, and occlusion ordering.
- Surpasses state-of-the-art layout models: 1.2–3× trajectory error drop, 2× rigid motion consistency, 1.5–2× occlusion accuracy.
Why It Matters
Enables directors to author complex multi-object video with simple 3D blocks, reducing manual labor.