New benchmark tests if AI agents can explain transformer circuits

Mechanistic interpretability has advanced in localizing circuits within neural networks, but explaining what those components actually do remains painstakingly manual. A new paper by Khan et al. tackles this bottleneck by asking whether language model (LM) agents can automate circuit explanation. The authors introduce AgenticInterpBench, a benchmark built from 84 semi-synthetic transformer circuits with 163 component-level annotations, and propose HyVE (Hypothesize, Validate, Explain), an agentic explainer that iteratively observes component behavior, generates hypotheses, and validates them causally. HyVE produces both component-level explanations and a holistic circuit-level task description. Tested across four different LLM backbones, HyVE recovers useful explanations in many cases, but no single backbone performs uniformly best. The analysis reveals that strong backbones tend to form observation-grounded hypotheses, while failures typically emerge later in the validation loop—often due to incomplete validation plans, code execution errors, or unresolved hypotheses.

To check real-world applicability, the team conducted a case study on an arithmetic circuit in Llama-3-8B, showing that HyVE's formulation extends beyond semi-synthetic circuits to naturally trained models. Overall, the work demonstrates that LM agents are promising as circuit explainers, automating parts of the interpretability pipeline. However, reliable validation—ensuring that the agent's hypotheses are correctly tested with minimal human oversight—remains the key obstacle. The paper suggests that future research should focus on improving agentic validation mechanisms, perhaps by integrating formal verification or better execution environments. For the AI safety and alignment community, these results offer a concrete path toward scaling up interpretability efforts, moving from manual inspection to semi-automated analysis.