MAD-HOI generates diverse hand-object interactions from text with variable-length control
New model handles composite motions, completion, infilling, and end-of-motion all from one objective
MAD-HOI, introduced by Ananya Bal and six co-authors, tackles a core limitation in text-driven hand-object interaction (HOI) generation. Previous diffusion models require pre-specified motion lengths and only work on atomic motions, while autoregressive approaches rely on discrete motion codes that lose contact-sensitive detail. MAD-HOI instead encodes hand and object motions into a continuous latent space while keeping the two streams disentangled, preserving fine-grained physical contact information and enabling separate control over each hand.
The architecture uses a masked autoregressive transformer to predict context features that condition a flow-matching head. This lets the model handle a full range of tasks from a single training objective: generating atomic motions, creating composite sequences, completing and infilling partial motion trajectories, and predicting End of Motion (EOM) to reliably terminate generation. On the ARCTIC and GRAB benchmarks, MAD-HOI outperforms other open-source methods in both diversity and physical plausibility, suggesting it could serve as a more flexible backbone for animation, robotics, and VR applications.
- Combines masked autoregressive transformer with flow-matching in a disentangled continuous latent space
- Supports variable-length generation, composite sequences, completion, infilling, and EOM prediction from one objective
- Beats baselines on ARCTIC and GRAB benchmarks for diversity and physical plausibility
Why It Matters
MAD-HOI makes text-driven hand-object animation more flexible and physically realistic, benefiting VR, robotics, and automated content creation.