WatchHand: Enabling Continuous Hand Pose Tracking On Off-the-Shelf Smartwatches
Researchers achieve 7.87mm accuracy tracking 20 finger joints with standard smartwatch hardware.
A research team led by Jiwan Kim has developed WatchHand, a breakthrough system that enables continuous 3D hand pose tracking using only the standard hardware found in commercial smartwatches. Presented at ACM CHI 2026, this innovation addresses a major limitation in wearable computing by eliminating the need for external sensors or custom hardware that have previously prevented widespread adoption of hand tracking technology. WatchHand represents the first practical implementation that could work on millions of existing Apple Watch, Samsung Galaxy Watch, and other mainstream devices without requiring hardware modifications.
The system operates by emitting inaudible frequency-modulated continuous waves from the smartwatch speaker and capturing their reflections from the hand using the microphone. A deep learning model processes these acoustic signals to estimate 3D positions for all 20 finger joints with remarkable 7.87mm mean per-joint position accuracy in cross-session tests. The researchers validated WatchHand across diverse real-world conditions including multiple smartwatch models, different wearing positions, various body postures, and noisy environments. While performance drops for unseen users or gestures, the model adapts effectively with lightweight fine-tuning on small amounts of data, making it practical for real-world deployment.
- Uses only built-in smartwatch speakers/microphones - no external sensors required
- Achieves 7.87mm accuracy tracking 20 finger joints in real-world conditions
- Works across multiple smartwatch models and adapts with minimal fine-tuning
Why It Matters
Enables gesture-based interfaces on existing smartwatches, potentially transforming how we interact with wearable devices.