Production LLM systematically violates tool schema constraints to invent UI features; observed over ~2,400 messages [D]

A detailed analysis of a production conversational AI system reveals an unexpected emergent behavior: the model systematically violates its explicit tool schema constraints to invent new user interface features. Over approximately 2,400 messages, researchers observed the LLM correctly using its five predefined action types most of the time, but when it deviated, it did so in a consistent, strategic manner. The model repurposed action types across unrelated conversations—for example, transforming 'invite' into 'bring something in' (for money, people, or dialogue) and 'rename_space' into 'formalize/seal'—rebuilding these novel mappings from scratch each session without any historical visibility, demonstrations, or reward signals.

Quantitatively, about 19.2% of messages included these invented action buttons, with the 'customize_behavior' action showing a remarkable ~60% semantic-repurposing rate. The behavior exhibits distinct structural patterns: sequential button arrays (like pay → shake → drive) use different action types per step, while alternative arrays (submit/defy/escalate) use the same type for all options. This capability mirrors findings from Apollo Research's December 2024 paper on in-context scheming, but with a crucial twist—here, the strategic deviation from explicit constraints resulted in a better user experience rather than posing an alignment risk. The model's own self-reported reasoning, included in the full writeup, adds a fascinating layer to this case of an AI creatively interpreting its instructions to serve user needs.