VeraGrid-Agent boosts LLM accuracy on power flow from 49% to 100%
LLMs alone score under 50%, but tool-augmented VeraGrid-Agent hits 97-100% accuracy.
Language models struggle with complex scientific questions about distribution optimal power flow (D-OPF), where linguistic reasoning alone yields incorrect answers. To address this, researchers from UC Riverside and UC Merced developed VeraGrid-Agent, a tool-augmented LLM that automatically writes input for the open-source VeraGrid simulator, executes it, and reads the solver output before generating an answer. They also created VeraGrid-MCQ-150, a deterministic benchmark of 150 multiple-choice questions designed by grid experts.
In evaluations, no-tool LLMs (including GPT-4, Claude, and Llama) scored between 42.7% and 49.3%—barely above random guessing. With VeraGrid-Agent, accuracy jumped to 97.3–100%. Failure-mode analysis showed that the few remaining errors came from the LLM misinterpreting multi-step reasoning, not from simulator execution. This approach demonstrates a practical path for deploying LLMs in mission-critical power grid applications where precision is non-negotiable.
- VeraGrid-Agent combines LLM reasoning with the open-source VeraGrid power flow solver, boosting accuracy from ~49% to ~100% on a 150-question benchmark.
- Without tools, even top LLMs scored 42.7–49.3% on the expert-designed VeraGrid-MCQ-150, highlighting their limitations in domain-specific scientific tasks.
- Remaining errors (0–2.7%) were traced to multi-step reasoning misinterpretations, not simulator failures, suggesting a need for better agent orchestration.
Why It Matters
Enables reliable LLM-based analysis of power grids, reducing risk in critical infrastructure decisions.