LLMs detect hallucinations perfectly but can't steer away from them

Researchers from multiple institutions tested a hidden assumption in mechanistic interpretability: that the direction in a model's activations which detects a behavior is the same as the direction which controls it. Using Gemma 2-2B-it, they found that for hallucination detection, the model achieves perfect separability (AUC=1.0 from layer 5) – it knows when it's hallucinating. However, the direction that best detects fake entities sits at a cosine of just 0.12 (about 83 degrees) from the direction that actually causes a refusal response. This means even though the model 'knows' a fact is fake, it cannot use that knowledge to steer its output.

The finding is not isolated: across four models from three families and scales 1B-9B, the cosine stays in [0.12, 0.20], and it remains almost identical before and after instruction tuning (0.1197 vs 0.1200), suggesting the gap originates in pretraining. A 15-degree rotation toward the refusal direction partially bridges the gap, achieving 73% and 60% refusal on two held-out fake-entity categories at 1.8% false positives. But crucially, the cosine does not predict steerability – detection is a high-dimensional class, not a single direction. The paper concludes that the cosine provides a weight-computable signature of the dissociation between knowing and steering, but is not a predictor of it.