AI Study Reveals How We Point at Screens Without Realizing
New research explains why you say 'this' and 'that' when using computers — and why it matters for AI.
People communicate how things should move by combining words with demonstrations: "open it like this." We present an interaction technique that brings this expressive resource to conversational 3D authoring. Building on "Put-That-There," our system combines speech, pointing, and spatiotemporal demonstrations to specify editable behavior.
A hand movement supplies evidence for a mechanism's axis, pivot, range, and pace; an animated preview makes the interpretation inspectable. Users refine behavior through further words or demonstrations, and the system can request a demonstration to clarify intent.
In a twelve-participant study comparing three input configurations, our system achieved 89% participant-declared completion versus 47% with speech alone, had the highest observed match rates on six categorical accuracy measures and tied on two, and was preferred by nine participants. This work makes demonstration part of an ongoing authoring conversation: behavior can be shown, inspected, and revised.
- The study is about how we use words like 'this' and 'that' when pointing at screens.
- Better understanding could lead to AI that follows voice and touch commands more accurately.
- No immediate product — it's early research shared on arXiv, a preprint server.
Why It Matters
Future AI could understand your gestures and words better, making technology more intuitive and less frustrating.