HVG-3D: Bridging Real and Simulation Domains for 3D-Conditional Hand-Object Interaction Video Synthesis

A research team led by Mingjin Chen has introduced HVG-3D, a novel framework that significantly advances the synthesis of hand-object interaction videos by moving beyond traditional 2D control methods. The system employs a diffusion-based architecture augmented with a specialized 3D ControlNet module, which encodes detailed geometric and motion information from explicit 3D representations. This architectural choice enables the model to perform explicit 3D reasoning during video generation, addressing the spatial expressiveness limitations that have plagued previous approaches.

HVG-3D operates through a hybrid pipeline that constructs input and condition signals, allowing for flexible and precise control during both training and inference phases. During inference, the system requires only a single real image paired with a 3D control signal—which can originate from either simulation environments or real-world data—to generate high-fidelity, temporally consistent videos. This capability effectively bridges the gap between real and simulated domains, enabling the model to leverage the strengths of both data types.

The researchers validated their approach on the TASTE-Rob dataset, where HVG-3D demonstrated state-of-the-art performance across multiple metrics including spatial fidelity, temporal coherence, and controllability. The framework's ability to utilize both real and synthetic 3D conditional data represents a significant advancement over previous methods that were limited by their reliance on 2D control signals. This breakthrough has important implications for applications ranging from robotics training and virtual reality to automated content creation and human-computer interaction research.