AI Glasses That Fix Your Mistakes Right Away
Imagine glasses that spot your screw-ups and tell you how to fix them instantly
Errors are inevitable in procedural tasks, yet most AR guidance systems focus on step-by-step instruction delivery rather than helping users recognize and recover from mistakes. We present RegulAR, an AR task assistant for procedural error recognition and recovery.
RegulAR models task instructions as a hierarchical dependency graph and combines this structure with a Multimodal Large Language Model (MLLM) to interpret egocentric observations during execution. This enables RegulAR to track progress, identify deviations by error type, estimate their impact on later steps, and deliver appropriately salient interventions through an in-situ head-up display that visualizes task state and recovery guidance.
By making procedural structure explicit, RegulAR supports not only next-step guidance, but also reasoning about what went wrong, why it matters, and how users can get back on track. In a within-subject study (N=12), participants reported better task-structure understanding and recovery support with RegulAR than the MLLM-only baseline.
- AI glasses called RegulAR watch you work and spot mistakes as you assemble furniture or cook, using a visual map of the task.
- It doesn’t just tell you what to do next—it explains why a step matters and how to recover if you mess up.
- In a 12-person study, users said it helped them understand the task better and recover faster than regular instructions.
Why It Matters
This could turn frustrating DIY projects or cooking disasters into quick, fixable moments—saving time, money, and your sanity.