LLMs converge on same programs when mutating code, new study finds
93% of LLM-generated program mutations revisit previously seen structures, limiting exploration.
Researchers from Northwestern University analyzed how LLMs mutate programs in a domain-specific language across varying prompts, models, and stochastic replication. Without selection pressure, they found that LLM-based mutation consistently converged toward narrow attractor regions. Structural convergence was severe: in 87% of chains, over 93% of mutations revisited a previously seen structural form, with most variation confined to terminal substitutions in recurring templates. Cycle analysis revealed short cycles and self-loops dominating the transition structure, meaning the LLM effectively circles back to the same code patterns.
This convergence persisted across different prompt wording and model choices, suggesting the phenomenon is intrinsic to the LLM mutation pipeline. In contrast, a classical genetic programming subtree mutation operator explored far more diverse structures under identical conditions. The authors argue that LLMs' semantics-aware capabilities—while powerful—carry a systematic bias toward homogeneity that undermines open-ended exploration. Accepted to the GECCO '26 workshop, the findings have implications for any system using LLMs for program evolution, automated code generation, or self-improving AI agents.
- 87% of LLM-driven mutation chains converged to attractor regions in program space.
- Over 93% of mutations revisited a previously seen structural form, with variation limited to terminal substitutions.
- Classical GP subtree mutation did not exhibit comparable convergence, highlighting a bias unique to LLMs.
Why It Matters
LLMs struggle with truly novel code generation, potentially limiting their use in evolutionary computation and self-improving AI systems.