Chat Debugging study: LLMs aid circuit debugging, but visual reasoning and confidence issues remain

A new arXiv study, Chat Debugging: An Exploratory Study of Human-AI Collaboration to Debug Analog Circuits, investigates how undergraduates use public-domain large language models to troubleshoot malfunctioning analog circuits. Researchers John Hu and Andrew Ash analyzed chat logs from students debugging pre-determined buggy circuits on breadboards and printed circuit boards under exam and time pressure. Through thematic analysis, they discovered multimodal usage patterns—students pasting images, schematics, and measurements into chat—and found that off-the-shelf LLMs offered considerable domain knowledge and sensible debugging suggestions, often pointing to correct fault classes like biasing errors, capacitor failures, or op-amp configuration mistakes.

However, the study also identified significant gaps. LLMs struggled with 2D/3D image-based reasoning, frequently misinterpreting physical layouts or visual circuit details. Their tone of confidence was often unjustified, leading students astray when the model was wrong. Additionally, students' own deficits in fundamental concepts and critical thinking amplified these errors—they accepted LLM suggestions without verifying, and foundational knowledge gaps prevented them from catching mistakes. The paper, accepted for the 2026 IEEE Frontiers in Education Conference (FIE), suggests human-AI collaborative debugging is promising but highlights the need for better multimodal reasoning, calibrated confidence, and stronger technical fundamentals in both AI and engineering education.