MUCOCO: Automated Consistency Testing of Code LLMs

A team of researchers has introduced MUCOCO, a novel automated framework designed to test the consistency of Code Large Language Models (LLMs). The core problem it addresses is that models like GPT-4, Claude 3, or Code Llama can produce different—and sometimes incorrect—outputs for programs that are semantically identical. Traditional benchmarks are static and hand-crafted, failing to systematically target this consistency property. MUCOCO automates the discovery process by applying semantic-preserving mutations (e.g., renaming variables, reordering statements) to create equivalent program variants, then checks if the LLM's generated code for these variants behaves differently than for the original.

In their evaluation, the researchers applied MUCOCO to seven LLMs across four distinct coding tasks. The results were striking: approximately 15% of the mutated inputs successfully exposed inconsistent behaviors from the models, such as generating code that produced different outputs or failed tests. MUCOCO significantly outperformed the closest existing baseline method, TURBULENCE, in its ability to uncover these flaws. This work shifts the focus from just measuring performance on static benchmarks to actively testing for fundamental reliability properties, revealing a hidden layer of instability in AI coding tools that developers rely on daily.