Editing One Neuron Fixes Gemma 4's 95% Repetition Loops

A new paper on arXiv reveals that Gemma 4 models (both dense and Mixture-of-Experts variants) have a reproducible failure: when asked to enumerate long factual lists (e.g., all 151 original Pokémon), they fall into tight verbatim loops or list decay into a single answer—occurring in up to 95% of cases. The authors, led by Aristotelis Lazaridis, traced this pathology to a small set of MLP neurons using per-layer ablation and attribution. In the smallest model (E2B), a single sign-inverted weight edit suffices to break the loop; larger models require editing a handful of routed experts. The edits preserve general-purpose benchmark scores and work across prompt rewording and inference-engine changes.

However, the surgery is not a panacea. The paper distinguishes 'simple loops' (repetition of the same token/answer) from 'doom loops'—where the model self-corrects in circles over a fact it cannot recall, exhausting the generation budget without committing to an answer. While the edits reduce doom loop frequency, they cannot eliminate them because the root cause is a knowledge gap, not a circuit bug. The researchers argue this is a fundamental limitation: weight editing can delete a loop, but it cannot supply a missing fact. For professionals, this means targeted fixes exist for specific failure modes, but robust long-form generation still requires better knowledge grounding.