The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning

A team of researchers from Carnegie Mellon University has published a significant paper titled 'The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning.' The study introduces a rigorous 'diagnose-measure-bridge-treat' framework to analyze a critical failure mode in modern large language models (LLMs). Through causal-behavioral analysis of problems like the 'car wash problem,' the researchers discovered that models rely on approximately context-independent sigmoid heuristics, where a salient surface cue (like distance) can exert 8.7 to 38 times more influence on the model's output than the actual goal of the task. Token-level attribution showed patterns more consistent with simple keyword associations than with compositional, step-by-step reasoning.

To measure the scope of this problem, the team created the Heuristic Override Benchmark (HOB), comprising 500 instances spanning 4 heuristic and 5 constraint families. The benchmark tests models with minimal pairs and gradients of explicitness. Results across 14 models, including GPT-4, Claude 3, and Llama 3, were stark: under a strict evaluation requiring 10 out of 10 correct answers, no model exceeded 75% accuracy. 'Presence' constraints (e.g., an object must be present to perform an action) were the hardest, with models achieving only 44% accuracy. Crucially, a minimal hint emphasizing the key object recovered an average of +15 percentage points, indicating the failure lies in inferring the constraint, not a lack of knowledge. Surprisingly, 12 out of 14 models performed worse when the problematic constraint was explicitly removed, revealing a conservative bias in their reasoning.

The research also tested interventions. Parametric probes confirmed the sigmoid heuristic pattern generalizes to cost, efficiency, and semantic-similarity scenarios. However, a promising mitigation was found: goal-decomposition prompting, which forces the model to enumerate preconditions before answering, recovered +6 to +9 percentage points in performance. The paper concludes that 'heuristic override' is a systematic and generalizable vulnerability in current LLM reasoning, providing both a diagnostic benchmark and initial pathways for treatment.