Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads
New study reveals specialized attention heads in models like Qwen-2.5 and Llama-3.1 that are critical for cross-lingual Chain-of-Thought.
A team of researchers from UT Austin and other institutions has published a new paper titled 'Bridging Latent Reasoning and Target-Language Generation via Retrieval-Transition Heads,' providing a breakthrough in understanding how multilingual large language models (LLMs) work. The study investigates attention heads in Transformer models, specifically identifying a previously unknown subset called Retrieval-Transition Heads (RTHs). These heads are responsible for governing the critical transition from internal reasoning to generating output in a specific target language, a key component for effective multilingual Chain-of-Thought (CoT) reasoning. This discovery moves beyond the known concept of 'retrieval heads' and isolates the precise mechanisms that enable models to answer complex questions across languages.
The research, tested on the Qwen-2.5 and Llama-3.1 model families across four multilingual benchmarks (MMLU-ProX, MGSM, MLQA, and XQuaD), demonstrates that RTHs are more vital for performance than standard retrieval heads. Experimentally masking these RTHs induced a significantly larger performance drop, proving their specialized role. This work provides a more granular map of a model's 'circuitry,' showing that retrieval heads are often shared across languages, but RTHs are the specific gatekeepers for language output. This mechanistic understanding is a major step toward debugging, improving, and potentially steering the reasoning processes of future multilingual AI systems, making them more reliable and interpretable.
- Identified new 'Retrieval-Transition Heads' (RTHs) that control the shift to target-language output in multilingual LLMs.
- Proved RTHs are more critical than standard retrieval heads; masking them caused bigger performance drops on 4 benchmarks.
- Tested on Qwen-2.5 and Llama-3.1 models, advancing the mechanistic understanding of multilingual Chain-of-Thought reasoning.
Why It Matters
This mechanistic discovery helps debug and improve multilingual AI reasoning, leading to more reliable and interpretable models for global applications.