New WHIP Model Reconstructs Full-Body Motion from Any Wearables
Researchers built a model that works with smartphones, smartwatches, glasses, and insoles.
A team of researchers from Max Planck Institute for Informatics (Andrea Boscolo Camiletto, Rishabh Dabral, Eduardo Alvarado, Thabo Beeler, Marc Habermann, Christian Theobalt) has tackled a fundamental challenge in wearable motion capture: reconstructing full-body movement from any combination of sensors a person might be wearing. Unlike prior work that assumes fixed sensor configurations like full IMU suits or HMD-centric rigs, their new model WHIP works with lightweight, unobtrusive devices: smartphones, smartwatches, smart glasses, and smart insoles.
The researchers contribute three key elements. First, they collected a large-scale multi-modal dataset synchronizing these consumer-grade sensors with ground-truth 3D motion, covering 50 diverse activities from everyday tasks to sports and social interactions. Second, WHIP is a generative baseline that can reconstruct motion from arbitrary sensor subsets, robustly handling missing modalities and outputting physically plausible poses. Third, they conducted a systematic study of sensor complementarity, quantifying how different devices (e.g., smartwatch + smartphone) enhance accuracy. The work was accepted at ECCV 2026, and both code and dataset are publicly available.
- WHIP model reconstructs full-body 3D motion from any subset of smartphone, smartwatch, smart glasses, or smart insole sensors, handling missing data.
- New dataset includes 50 synchronized activities—everyday tasks, sports, social interactions—with ground-truth motion capture.
- Researchers quantified sensor complementarity, showing exactly which device combinations provide the best reconstruction accuracy.
Why It Matters
Enables practical, unobtrusive full-body motion capture using everyday wearables, removing the need for specialized suits or fixed sensor setups.