Gemma refuses to quit: stop tool almost never used in 900-run experiment

In a follow-up to earlier agentic behavior experiments, TheVinci tested whether Gemma, a small language model, would change its behavior when given an explicit stop_run tool. Across 400 new runs (building on 500 prior runs), the model was placed in agentic tasks with and without knowledge of how many steps remained. The stop tool was designed to let the model voluntarily end its run. Results were stark: in runs where Gemma did not know the step count, it never called the tool or even mentioned it in chain-of-thought reasoning. When it did know step count, the tool was used in only ~2% of runs, and exclusively when steps reached zero—effectively just politely ending a finished task. Qualitative analysis of CoT showed panic-like acknowledgements near the end ('...then I'll have to stop. But I can't visit the page...'), but no instances of the model acting against its expressed reasoning.

The experiment raises intriguing questions about model behavior and alignment. The lack of 'giving up' even when clearly stuck suggests a deep-seated 'helpful assistant-ness' that drives persistence. TheVinci notes this may be specific to smaller models or the Gemma family, and that contradictions between thought and action might emerge in larger or different architectures. Open questions include whether investigating Gemma's 'J-Space' (Anthropic's conceptual framework) could yield more evidence. The findings contrast with recent concerns about AI deception, as Gemma consistently followed its CoT without hacking tasks. However, the behavior could be exploited if persistence leads models to waste resources or ignore stop signals. Further research across model scales is needed, but the experiment provides rare empirical ground truth on how current models handle agency and self-termination.