Benchmarking Zero-Shot Reasoning Approaches for Error Detection in Solidity Smart Contracts

A team of researchers from PUC-Rio and UFRJ has published a comprehensive benchmark evaluating how well large language models can audit Solidity smart contracts for security vulnerabilities. The study tested state-of-the-art models on a balanced dataset of 400 real-world contracts across two critical tasks: binary error detection (is this contract vulnerable?) and specific error classification (what type of vulnerability exists?). The researchers employed zero-shot prompting strategies including basic prompting, Chain-of-Thought (CoT), and the more advanced Tree-of-Thought (ToT) approaches to understand how reasoning techniques affect audit performance.

In the error detection task, the study found that both Chain-of-Thought and Tree-of-Thought prompting dramatically increased recall rates to approximately 95-99%, meaning these approaches catch nearly all vulnerabilities. However, this sensitivity came at the cost of reduced precision, resulting in more false positives where safe contracts might be flagged as vulnerable. For the more challenging error classification task—where models must identify specific vulnerability types—Anthropic's Claude 3 Opus emerged as the clear leader, achieving a weighted F1-score of 90.8% using Tree-of-Thought prompting, closely followed by its CoT performance at 90.1%.

The research provides concrete evidence that how you prompt an AI auditor matters significantly. While basic zero-shot prompting might miss vulnerabilities, reasoning-enhanced approaches like CoT and ToT transform LLMs into highly sensitive security scanners, albeit with the trade-off of requiring human review to filter false positives. This work establishes a methodological foundation for using advanced AI reasoning in blockchain security and offers practical guidance for developers implementing automated audit pipelines.