ProcAgent runs on-device AI guidance for assembly tasks in 8 seconds

ProcAgent is a fully on-device, agentic, vision-based assistant designed to guide users through procedural tasks like furniture assembly and home repair. Developed by researchers from the University of Texas at Dallas and others, it runs on a single NVIDIA Jetson AGX Orin edge device. Unlike prior multimodal assistants that rely on cloud inference and always-on perception, ProcAgent uses a propose-and-verify architecture: it continuously proposes user progress from camera input, but only invokes expensive vision-language reasoning when ambiguity or deviation is likely. This keeps latency low while maintaining accuracy. A symbolic task graph tracks progress, and an LLM-based interaction agent handles proactive interventions and question answering with human-in-the-loop confirmation.

The system achieves impressive responsiveness despite running entirely on-device: text-only queries resolve in approximately 2 seconds, and visually grounded queries in around 8 seconds. In a user study with 10 participants completing assembly tasks, ProcAgent received positive ratings across comprehensibility, actionability, and privacy comfort. These results demonstrate that adaptive, real-time procedural assistance can be delivered on edge hardware without sacrificing usability or privacy. The work opens the door for privacy-sensitive, latency-critical applications in domestic settings where cloud connectivity is undesirable or unavailable.