New benchmark tests LLM agents on power grid contingency planning

PowerAgentBench-SS, introduced in a June 2026 arXiv paper by Mylonas et al., is the first benchmark specifically designed to evaluate tool-using LLM agents on real-world power system steady-state engineering workflows. Unlike traditional benchmarks that test numerical solvers, prediction models, or sequential controllers in isolation, this framework requires an agent to complete an end-to-end task: inspect a grid case, select appropriate tools, call power-flow simulators, screen for N-2 contingencies, propose admissible mitigations, validate results, and produce an auditable evidence trail. The benchmark exposes public case data (using deterministic variants of the IEEE 39-bus system), action constraints, a tool API, and a validation budget to the agent, while a hidden evaluator recomputes physical validity and scores the submitted report using risk-sensitive metrics.

The paper defines a detailed agent interface, tool contract, evidence log, and metrics such as submitted recall, evidence-backed recall, false-safe penalties, severity regret, residual violation score, action cost, tool-use efficiency, and workflow diagnostics. In a concrete instantiation, the authors ran a DC thermal N-2 contingency-search pilot with scripted baselines, an LLM JSON-command adapter, three locally hosted Ollama LLM agents, and one OpenAI API agent. Results showed that agents are distinguished not just by top-contingency discovery but also by validation-budget discipline, explicit submission behavior, type coercions, duplicate validations, evidence-backed reporting, and mitigation actions. The work underscores that solver-only or answer-only evaluations miss critical failures – such as budget overruns, weak evidence trails, or unsafe mitigation proposals – and calls for richer benchmarks that capture the full agentic workflow.