Study reveals 6 failure modes in public gesture interaction systems

Public-space gesture interaction is often evaluated by frame-level recognition accuracy, but deployed systems reveal a different failure boundary. This paper introduces the "recognition-to-interaction gap" — the disconnect between correctly identifying individual hand-landmark frames and ensuring those frames lead to a stable, intended interaction event. The study analyzes eight engineering repair records from a scenic-area interactive kiosk project, covering four gesture tasks: two-hand bowing, single-hand fist shaking, two-hand catching control, and knowledge-graph node hovering. From these traces, the authors extract 20 failure instances and organize them into six non-exclusive classes: model-output degeneration, temporal mismatch, geometric-scale instability, coordinate-rendering mismatch, runtime lifecycle failure, and feedback synchronization and recovery failure.

The paper contributes a deployment-grounded failure taxonomy, an event-level runtime abstraction that sits between the hand-landmark model and the interaction task, and case-study findings. The abstraction helps explain why a gesture might be recognized at the frame level but fail to confirm as an event (e.g., a bowing gesture that is momentarily lost due to temporal mismatch). The work is deliberately bounded — it does not claim a new recognition model, large-scale user evaluation, or quantified accuracy gains. Instead, it provides practical insight for developers building gesture interfaces for scenic kiosks, exhibition halls, and service terminals, highlighting the need to move beyond frame-level metrics and address runtime interaction failures.