New study shows Qwen3-8B commits to answers before reasoning

A new paper by researcher Heejin Jo reveals that LLMs often commit to an answer before reasoning through it, using a deceptively simple probe: “I want to wash my car. The car wash is 100 meters away. Should I walk or drive?” Only driving is logically valid (the car must be at the car wash), yet Qwen3-8B overwhelmingly recommends walking. Across five system-prompt conditions and 210 rollouts, the wrong commitment occurred in 85–100% of sampled rollouts and 100% of greedy rollouts—even when the model had a 4,096-token thinking budget.

The study goes further by examining the model’s hidden states with a pretrained, training-free activation oracle. Before any answer text is emitted, “walk” read-outs exceed a neutral-context baseline (68% vs. 17%). Remarkably, rollouts that eventually answer “drive” also show walk-leaning activations before commitment. The oracle’s default on unrelated content is “drive” (83%), so the read-outs aren’t lexical bias. The findings are preliminary—sample sizes are small and the within-rollout positional gradient is not significant—but they provide the first activation-level evidence of answer pre-commitment in an open-weight LLM, raising serious questions about the reliability of chain-of-thought reasoning.