Qwen3-14B study shows linear probes detect task format, not reasoning type

A new study by Sahoo, Jain, Chadha, and Chaudhary (accepted at Trustworthy NLP Workshop, ACL 2026) challenges a core assumption in mechanistic interpretability: that linear probes of LLM hidden states can detect distinct reasoning modes. Using Qwen3-14B, they probed three reasoning types—deductive (LogiQA 2.0), inductive (ARC-Challenge), and abductive (αNLI)—and found that at layer 32 of 40, linear classifiers achieved 100% cross-validated accuracy with well-separated geometric representations (intrinsic dimensionalities: 20.6, 28.5, 33.6; convex hull contamination ≤1.5%). However, when the authors residualized confounding features such as source identity, number of options, and response length, classification accuracy fell to chance. Trace-anchor similarity analysis showed 42.5% shared reasoning across tasks (vs. 33.3% chance), and causal steering with random controls revealed no functional link between geometry and reasoning mode (p=0.286). The findings imply that earlier claims of LLMs learning distinct reasoning representations may be artifacts of task format rather than computational structure.

The paper's central contribution is a methodological warning: linear probes are highly susceptible to format confounds, and high accuracy does not guarantee that the model is reasoning differently. The authors recommend that future mechanistic interpretability work routinely deconfound features like task source, option formatting, and response length. For LLM researchers and engineers, this means re-evaluating interpretability claims that rely solely on probe accuracy. It also underscores the need for more rigorous causal intervention methods—such as the random-control steering used here—to establish whether hidden-state geometry actually drives reasoning behavior. The study is a timely reminder that as LLMs become more capable, our tools for understanding them must become more sophisticated.