LLMs generate near-human quality properties for Solidity smart contracts

Smart contracts deployed on blockchains are immutable, making post-deployment bug fixes impossible. This amplifies the need for rigorous pre-deployment testing, typically done via property-based testing with manually written properties—a slow, expert-dependent process. Researchers explored whether state-of-the-art LLMs could generate these properties automatically, using mutation testing to measure quality. They tested various models (Gemini Pro 1.5, GPT-4, etc.) with zero-shot, few-shot, and prompt-chaining techniques.

Results showed Gemini Pro 1.5 with prompt chaining scored highest on average (25.99% mutation score), approaching human experts (31.75%). Performance varied significantly per contract: on the LibBit contract, the LLM scored 74.34% vs. human 74.83%—essentially expert-level. This indicates that while average performance still lags, specific cases demonstrate LLMs can match human quality. The study highlights prompt-chaining as a key technique and suggests further improvement with fine-tuned models or larger context windows.